Tabular is a well know approach to create a data model that can be then published and analysed in Analysis Services. Deploy a new feature on our tabular model is very straightforward, but what if we want to check the change we made before we deploy our model? ALM Toolkit is a great tool for that and more! In fact, it does not only enable us to deploy our model with a variety of options, but it also allows us to compare the datasets and see the difference between our published data model and the data model we're working on locally. We see the change in form of metadata, as in the image below 👇 In this case, the data type of the "formatString" column has been modified. Hope that was an interesting piece of information!

Hi, I'm working on my first ever ETL pipeline project from zero, and I designed this architecture for aws. It is based on research I've done and chatting with more experienced colleagues. I'm sure it can be improved but so far I feel comfortable with it. Brief explanation of the data lake: - the "raw" bucket is intended as a landing zone, with data in its original state (for auditing purposes) - the "cleansed" bucket contains data after fixing inconsistencies, invalid values, deleting unnecesary columns, deduplicating, etc. - the "curated" bucket is intended to have the data ready to be ingested to the data warehouse, matching fact and dimension tables schemas. First of all, I'd like to know what you think in general. Then I have a couple of questions: 1. should I store the processed data in the cleansed bucket in csv or parquet would be a better option? 2. should I keep curated as an s3 bucket in parquet it would be better as a database schema? 3. Last but not least, I'm struggling with the pipeline strategy, meaning how would I perform incremental loads? (I plan to batch update daily). What techniques can I use? Any design considerations? AWS Glue jobs have a bookmarks feature to avoid reprocesing data, but if the schema is updated in the catalog, the job will fail, and process all data in the source. Thanks in advance. I couldn't think of a better place to share my concerns than this community.

If I could go back in time, these are the 6 things I wish I knew about Data Engineering. (It would have saved me a lot of wasted time and effort.) - 1. There will always new fires to put out Every new bug, urgent request, etc. feels like the sky is falling. And you'll stress out trying to complete it as soon as possible. ...Only to get hit with a new one the next week. Keep it in perspective. ============= 2. There's no universal "best" approach So much time gets spent debating decisions & strategies. It's vital to have an opinion and think through options. But most approaches typically work out as long as you're on the same page. I've seen many bridges burned due to an inability to compromise. ============= 3. Beware of personal scope creep It feels noble to fix every bug or inefficient query you come across. But this distracts from assigned work & sends you down an unnecessary rabbit hole. Instead, log these issues separately and complete the task at hand first. ============= 4. Show, vs tell, new solutions Don't expect a change to happen by just casually explaining it. People are busy and have other things on their mind. Take time to put together a sample of your solution that you can show. Once people see it, then they pay attention. ============= 5. Curiosity & continuous learning are critical This is a technology career and it constantly evolves. If you become stagnant, so will your career. ============= 6. Every company's data is a mess, you're not alone The grass isn't always greener on the other side. Also, most problems are human problems vs tech/data problems. ============= In short, Data Engineering is a journey, not a destination. And we are all on this journey together. What are some lessons you've learned in your career so far?

I tend to waste a lot of time over analyzing things - especially when it comes to new content. So rather than continue to do that, this time I figured I'd share a raw behind-the-scenes look at something new I'm working on directly with you. Your feedback can make sure it's on the right track and will help you overcome real challenges you're facing. (or let me know if it's off the mark) Rather than bore you with text, I've recorded a Loom video breaking it down. Heads up - this was a completely unplanned video and I ramble a bit. But hopefully the main points still come across. Any feedback at all on this would be incredibly valuable. Feel free to leave a comment here or DM me directly if you'd prefer to remain private. Thanks as always!

It's one thing to read articles or watch videos about perfectly crafted data architectures and think you're way behind. But back here in reality, things get messy & nothing is ever perfect or 100% done. Most of us are usually working on architectures that are: - Old & outdated - Hacked together - Mid-migration to new tools - Non-existent Or perhaps you're one of the lucky ones that recently started from scratch and things are running smoothly. Regardless, the best way to learn what's working (and not working) is from others. I believe this could be one of the best insights this community can collectively offer each other. So let's hear it. What does your data stack look like for the following components? 1. Database/Storage 2. Ingestion 3. Transformation 4. Version Control 5. Automation Feel free to add other items as well outside of these 5, but we can focus on these to keep it organized.