Data movement between workspaces · Learn Microsoft Fabric

Data movement between workspaces

Hopefully I am not over complicating this but we are moving from Silver to Gold and want to make sure thinking through this correctly. Our Silver is Lakehouse and Gold is Warehouse and we have in 2 separate workspaces for security purposes. One example I have here is Company table. We have multiple sources with company and need to combine them together. Source A and B may have same companies as well as different ones so we have a another company table that combines the sources via a notebook. Moving the combine company data table that is in silver to gold and only doing the changes (some tables are quite large) had 2 thoughts. 1) In Silver in the combine table having an action column that is INSERT, UPDATE or PROCESSED. We could easily now identify what needs to be inserted and updated and use a pipeline to move. This has the assumption we have same data. My other thought 2) was creating a lakehouse is gold to act as staging and have short cuts to the silver lakehouse merged company table, then can run a merge notebook for gold warehouse and this makes sure we are updating everything. This option has the extra lakehouse, shortcuts we need to keep in sync though.

Was curious on thoughts on these 2 options as well as new options that I am not thinking of, always learning.

20 comments