Delay between a lakehouse and it SQL analytics endpoint

Hi All,

A know issue is the fact that when I ingest data into a lakehouse, there can be (nah will be :)) a delay before that data is actually available in the SQL analytics endpoint.

Up until now I have been trying to tackle this by adding 5 or even 10-minute delays between lakehouse ingestion steps and addressing the SQL endpoint to do joins to dim tables, ETL etc. This works most the time but of course causes delays in running pipelines (and it is not 100% proof).

I came across the below MS article, has anyone tried this yet and is it really as simple as adding a script activity in the pipeline along the lines of:

"SELECT TOP(1) 1

FROM [lakehouse].[dbo].[tablename]"

Or should i read all rows, like SELECT 1 and not top (1) ?

https://learn.microsoft.com/en-gb/fabric/known-issues/known-issue-1092-delayed-data-availability-analytics-endpoint-pipeline?wt.mc_id=fabric_inproduct_knownissues

Cheers

Hans

1 comment