Hey! In my company we're on our way to release some public data to our users. I'd like to step up this and not only release the data via a public s3 but offer our users a tool to explore and aggregate this datasets easily. ¿Do you know if something like this exists? I've been looking but haven't found anything useful yet. Update: I'm looking for something like Dune Analytics but for general data (not scoped to blockchain as this project is.) Thanks!

Do you think its worth it to create templates like with cookie cutter in order to automate the create of basic modules, project structure etc? One issue with this approach in my opinion is that almost every project is different and many times even small differences in the user-case requirements may require big differences in the overall data workflow.

It's one thing to read articles or watch videos about perfectly crafted data architectures and think you're way behind. But back here in reality, things get messy & nothing is ever perfect or 100% done. Most of us are usually working on architectures that are: - Old & outdated - Hacked together - Mid-migration to new tools - Non-existent Or perhaps you're one of the lucky ones that recently started from scratch and things are running smoothly. Regardless, the best way to learn what's working (and not working) is from others. I believe this could be one of the best insights this community can collectively offer each other. So let's hear it. What does your data stack look like for the following components? 1. Database/Storage 2. Ingestion 3. Transformation 4. Version Control 5. Automation Feel free to add other items as well outside of these 5, but we can focus on these to keep it organized.

Hello! Welcome to The Modern Data Community. The goal of this community is to help Data Engineers on small (or solo) teams confidently build modern architectures by simplifying key concepts, clarifying common strategies & learning from others. Pumped to have you here! ==================== HOW IT WORKS ==================== By joining, you get instant access to tons of free content (see Classroom). Dive right in. But even more can be unlocked by contributing to the Community, which I encourage you to do. It works like this: Contribute (post/comment) >> Get points (likes) >> Reach new levels >> Unlock content ==================== 6 SIMPLE GUIDELINES ==================== ❌ Do not post error messages looking for others to debug your code. That's why Stack Overflow and other tool-specific Slack channels exist. ❌ Do not use this community for self-promotion (unless Admin approved). We all know it when we see it. ❌ Do not create low-quality posts with poor grammar/spelling or that provide little (or no) value. These will be deleted. You can do better! ✅ Ask questions, share your experiences & overall be a good person. This not only helps everyone get better, but can help you unlock bonus content faster. Win-Win. ✅ Speaking of wins, share yours! Whether it's finally solving a complex problem, hitting a team milestone or starting a new gig - post about it. You'll get the props you deserve and it just might inspire somebody else. ✅ Take the time to craft thoughtful posts & always proof-read before hitting submit. We're all about quality here. High quality posts --> more engagement (aka you'll climb the leaderboard & unlock content) --> ensures the community stays enjoyable for everyone. ==================== QUICK LINKS ==================== Here are a few links to help you get going: - Classroom - What's Your Data Stack? - Leaderboard - Work with me (Kahan Data Solutions)