ELT / ETL Orchestration Framework · Learn Microsoft Fabric

ELT / ETL Orchestration Framework

Question: Best Practices for Hosting Orchestration Metadata in Fabric in a Metadata-Driven ELT/ETL Framework

I am currently tasked with creating a metadata-driven ELT/ETL orchestration framework. I am evaluating different options for hosting orchestration metadata and would appreciate insights and recommendations from the community.

Here are the options I've considered:

Azure SQL
Lakehouse (.delta format)
Warehouse (.delta format)

In my experience, Azure SQL has demonstrated better performance due to its transactional operations. However, I encountered concurrency issues with the Lakehouse and Warehouse approaches. I managed to mitigate these issues by partitioning the Delta tables based on the columns used in the update statement predicates, though I noticed some performance lag.

Previously, while using Synapse, the metadata was stored in Azure SQL. Now, as we transition everything to Microsoft Fabric, I am seeking advice on the best practices for this setup.

What solutions are others implementing, and what recommendations can you provide for hosting orchestration metadata in this new environment?

Thank you in advance for your suggestions!

1 comment