What is the ratio of people in your organization who are working on your data integration layer vs your data modeling/transformation layer vs BI/Visualization layer? Regardless of job title, be it data engineer, analytics engineer, data analyst, reporting analyst, whatever, how many people in your stack are working just on data integration vs modeling? I ask because I've seen this fluctuate in the orgs I'm in based on tooling choice and the health of the system. In the best, most agile org, the ratio of data engineers to analytics engineers was 1:3 to 1:4. In the least agile, most frustrating (for the business stakeholders waiting on data and reports), the ratio was 3:1 to 4:1, totally reversed. Thus far, this has been a tooling issue. The choice of data integration tool caused an enormous amount of manual work to acquire data and put together all the orchestration pieces etc before it was available for modeling or transformation. Help me fight my bias and tell me what you've seen in your previous roles and in your current role

Hey team! One of my companies goal the following quarter will be to define an aggregated metric layer for rapid insight of business important metrics. I've done something similar previous but not with dbt. I'd like to know if anyone has done this with dbt core and how would be the best way to approach it. Thanks!

Hey! In my company we're on our way to release some public data to our users. I'd like to step up this and not only release the data via a public s3 but offer our users a tool to explore and aggregate this datasets easily. ¿Do you know if something like this exists? I've been looking but haven't found anything useful yet. Update: I'm looking for something like Dune Analytics but for general data (not scoped to blockchain as this project is.) Thanks!

Hello everyone, I'm reaching out to see if anyone here has experience using Orkes for data pipeline orchestration. In my organization, our application team has recently adopted Orkes for application workflow automation. They have tasked us with evaluating whether Orkes can also be effectively used for ETL (Extract, Transform, Load) pipeline orchestration. Our current setup involves orchestrating ETL pipelines using Azure Data Factory (ADF). I'm trying to compare same pipeline implementation using Apache Airflow, and now potentially Orkes. Our goal is to compare these tools and identify the best fit for our needs. I have been researching ETL pipeline implementations with Orkes but haven't come across any concrete examples or case studies. I'm particularly interested in understanding if Orkes can handle our specific ETL pipeline requirements, which include: 1. Validating file schemas received from clients. 2. Triggering Databricks workflows for data cleaning. 3. Staging cleaned data in SQL Server. 4. Loading staged data into application tables via SQL stored procedures. If anyone has experience or insights into using Orkes for similar ETL processes, I would greatly appreciate your sharing. Any example workflows, implementation details, or general advice on using Orkes for data pipeline orchestration would be incredibly helpful as we make our decision. Thank you in advance for your assistance!

Hey team! I'm working in building a new dbt project on our snowflake datawarehouse. The idea is to improve how models are built, and the data architecture overall. One of the things I believe will take us some time is re-arranging all the existing metabase questions and migrating them from old models to the new ones. Anyone has faced this challenge before? If not, what do you thing would be the best way to do it?