Rollback and Retry on the pipeline failure

Aug '24 (edited) • 🤓 Tech Chat

Hello,

My pipeline is getting heavier - against the "best practice" that I read before......

The pipeline is built using the medallion approach - bronze, silver and gold in the lake house and the final table at the data warehouse. Also, several management table for pipeline management, incremental data load and table metadata management.

When the pipeline got some exception or failure during the process, the data should be rollbacked from several phase tables - I have to put "clean-up activity" on the failure to delete the data that the pipeline populated in the previous activities so that the pipeline can be rerun without any issue.

Is this what you are doing as well when you build the medallion data pipeline? Any idea to improve the processes?

0 comments