Incremental Refreshes in Dataflow · Learn Microsoft Fabric

19d (edited) • Technical

Incremental Refreshes in Dataflow

Hello, i wanted to know a couple of things about the incremental refresh in a Dataflow Gen2.

so i implemented an incremental refresh on my tables in a dataflow, i have 8 tables there.i set it with the following settings :

extract data from the past 1 week, bucket size is 1 day and only extract data when lastmodifieddatetime value changes. the tables usually take around 30min to refresh without incremental refresh. when i ran the dataflow first time it took 30 minutes, but after the first time it still was taking 30 minutes? i am not sure if i am doing anything wrong but shouldn't the refresh time be less than the usual refresh time?

also one more thing, is it better to have a seperate dataflow for each table or is it ok if they're all in 1 dataflow?

thank you in advance!

Update :

the dataflow was refreshing and all processes went perfectly, but then the dataflow failed on a "writing to destination" activity for some of the tables with the below error :

There was a problem refreshing the dataflow: "The refresh for this entity couldn't be executed because the user has reached the evaluation quota. Try again later or consider reducing the overall evaluation usage of the user". Error code: 999999.

what does that impose to?

1 comment