Activity
Mon
Wed
Fri
Sun
Jun
Jul
Aug
Sep
Oct
Nov
Dec
Jan
Feb
Mar
Apr
May
What is this?
Less
More

Memberships

Modern Data Community

Public • 573 • Free

1 contribution to Modern Data Community
Exploring Orkes for ETL Pipeline Orchestration: Seeking Insights and Examples
Hello everyone, I'm reaching out to see if anyone here has experience using Orkes for data pipeline orchestration. In my organization, our application team has recently adopted Orkes for application workflow automation. They have tasked us with evaluating whether Orkes can also be effectively used for ETL (Extract, Transform, Load) pipeline orchestration. Our current setup involves orchestrating ETL pipelines using Azure Data Factory (ADF). I'm trying to compare same pipeline implementation using Apache Airflow, and now potentially Orkes. Our goal is to compare these tools and identify the best fit for our needs. I have been researching ETL pipeline implementations with Orkes but haven't come across any concrete examples or case studies. I'm particularly interested in understanding if Orkes can handle our specific ETL pipeline requirements, which include: 1. Validating file schemas received from clients. 2. Triggering Databricks workflows for data cleaning. 3. Staging cleaned data in SQL Server. 4. Loading staged data into application tables via SQL stored procedures. If anyone has experience or insights into using Orkes for similar ETL processes, I would greatly appreciate your sharing. Any example workflows, implementation details, or general advice on using Orkes for data pipeline orchestration would be incredibly helpful as we make our decision. Thank you in advance for your assistance!
1
2
New comment Apr 4
Thanks Michael. No issues/limitation with ADF. Trying to minimize number of tools and we have dynatrace configured in Orkes for observability.
1-1 of 1
Dineshkumar Gopalakrishnan
1
3points to level up
@dineshkumar-gopalakrishnan-3397
ETL Developer and Trying to get into Big data and AI.

Active 26d ago
Joined Dec 27, 2023
powered by