Need Help with Executing Warehouse SQL Query Transformations in Fabric Spark Notebook

Avinash Sekar

Aug '24 (edited) • Technical

Hi everyone,

I'm working on a project that involves multiple query transformations. Here's an example of one such transformation:

QUERY:

SELECT col1 AS name, col2 AS price FROM warehouse_name.schema_name.table_name1

UNION

SELECT col3 AS name, col8 AS price FROM warehouse_name.schema_name.table_name2

I need to execute these queries from a Fabric Spark notebook. This step is crucial for my validation criteria. Currently, I have 140 transformation queries that I need to validate by checking counts and performing manual difference calculations. This process is very time-consuming.

If I can query these transformations and store the results as a DataFrame, it will significantly simplify the validation process.

Could anyone guide me on how to achieve this in a Fabric Spark notebook? Any suggestions or best practices would be greatly appreciated.

Thanks in advance for your help!

15 comments