Variable delays in data - Lakehouse schema shortcuts

We are currently facing issues with intermittent, and sometimes very long delays in data coming through into shortcut sources (sometimes more than 1 day). we thought we had cracked it by forcing a meta data refresh in the lakehouse through a scheduled notebook by using this command spark.sql('REFRESH /'). However, I found this was not solving the issue today, nor did a manual meta data refresh, which may be because it was stuck or failing to refresh as the spinning icon remained spinning and I had to refresh the page. The schema enabled lakehouse is still in preview so it may just not be ready for production yet.

4 comments