Exploring Orkes for ETL Pipeline Orchestration: Seeking Insights and Examples

Hello everyone,

I'm reaching out to see if anyone here has experience using Orkes for data pipeline orchestration. In my organization, our application team has recently adopted Orkes for application workflow automation. They have tasked us with evaluating whether Orkes can also be effectively used for ETL (Extract, Transform, Load) pipeline orchestration.

Our current setup involves orchestrating ETL pipelines using Azure Data Factory (ADF). I'm trying to compare same pipeline implementation using Apache Airflow, and now potentially Orkes. Our goal is to compare these tools and identify the best fit for our needs.

I have been researching ETL pipeline implementations with Orkes but haven't come across any concrete examples or case studies. I'm particularly interested in understanding if Orkes can handle our specific ETL pipeline requirements, which include:

1. Validating file schemas received from clients.

2. Triggering Databricks workflows for data cleaning.

3. Staging cleaned data in SQL Server.

4. Loading staged data into application tables via SQL stored procedures.

If anyone has experience or insights into using Orkes for similar ETL processes, I would greatly appreciate your sharing. Any example workflows, implementation details, or general advice on using Orkes for data pipeline orchestration would be incredibly helpful as we make our decision.

Thank you in advance for your assistance!

2 comments

Modern Data Community

Public group

A community of data professionals building architectures with modern tools & strategies.

Leaderboard (30-day)