Activity
Mon
Wed
Fri
Sun
Jun
Jul
Aug
Sep
Oct
Nov
Dec
Jan
Feb
Mar
Apr
May
What is this?
Less
More
Memberships
Modern Data Community
Public ā¢ 568 ā¢ Free
22 contributions to Modern Data Community
Feb 27Ā
inĀ
Choosing Between BigQuery, Redshift & Snowflake
Hello everyone, I'm currently in the process of evaluating data warehouse solutions to centralize data from 2 transactional Mysql databases, Events data from Segment, Salesforce & Intercom. Our goal is to load this data into a warehouse, perform some dbt transformation & then connect it to a BI Tool. We're considering BigQuery, Redshift, and Snowflake as potential solutions but are having a tough time deciding which one would be the best fit for our needs. The main key considerations we're looking at are Cost, Ease of Use, Performance and Speed Any recommendation you guys have that might be useful!
1
12
New comment Mar 17
2 likes ā¢ Mar 6
I had the same question 3 months ago :) Here is how I approached. I made a comparison table with these three options, something like: Feature --- Redshift --- BigQuery --- Snowflake Then, for each feature I described how particular DWH behaves. There were features like "Supported data types", "Architecture", "Scalability", "Maintenability", "Pricing" etc. After applied medallion tiers (Gold, Silver, Bronze) for each feature and DWH. In the end, Snowflake won this competition :) Here is a short screenshot of my summary:
2 likes ā¢ Mar 6
@Tomas Truchly to put it differently, don't try to use data warehouse as operational database :)
Feb 28Ā
inĀ
SQLMesh v dbt
I've recently been looking into SQLMesh and wondering if anyone has any project experience and can comment on how it compares to dbt core in practice. Also if you've another transformation tool feel free to share your experience.
3
6
New comment Apr 3
0 likes ā¢ Mar 6
Seems like SQLMesh is gaining traction, but people are afraid to test-drive it on real projects :)
0 likes ā¢ Mar 6
@Tomas Truchly competition usually makes the market better. No one is going to switch to anything else unless it's gonna be level of magnitudes better. I like some ideas of sqlmesh, they will eventually make dbt better. (kinda already do)
Feb 12Ā
inĀ
What is the value of extracting data into a cheap file store?
I've often seen it recommended to extract data out of a data source into a parquet file into a cheap file storage like Azure Blob or s3 buckets... what is the value of adding this step when all I'm doing is copying it on into a SQL Database?
2
6
New comment Feb 19
2 likes ā¢ Feb 14
1. Backup purposes 2. If you gonna switch from SQL DB to Spark/Databricks, the migrations will happen easier. 3. In addition to previous point, you could run parallel data pipelines on your data using external tools, like Spark/Glue/Trino/etc.
Feb 10Ā
inĀ
Data Vault
Data Vault Modeling I just started to learn about this topic. Hub, Link, and Satellite. I am trying to understand the correlation with Kimball, or if it is an extension of it. How to effectively applied in real-world challenges
1
6
New comment Feb 14
2 likes ā¢ Feb 14
Never done DV myself. But after talking to a colleague who did it, it's clear that you need to fully understand the approach, otherwise you risk to blow your infra costs pretty dramatically (that's what happened to my colleague). Of course, Kimball won't save you from poor implementation, but imo because it's easier to understand, there are potentially less mistakes you can make.
Feb 14Ā
inĀ
Experiences with Microsoft Fabric
Hey everyone! I'm curious if anyone has experience with or has used Microsoft Fabric. My company is currently considering using Fabric as an all-in-one warehouse-solution for data transformation, modeling, and BI. While our use case is relatively straightforward, I was wondering whether Fabric or rather the integrated tools for engineering and modeling are truly sufficient to cover the entire workflow efficiently.
2
2
New comment Feb 14
1 like ā¢ Feb 14
Would like to hear the feedback as well. I think that if your company is already using Microsoft stack, Fabric should be a no-brainer. But wanna hear a real experience :)
1-10 of 22
@oleg-agapov-7263
Analytics/Data Engineer building Modern Data Stack solution
Active 37d ago
Joined Jan 5, 2024
powered by