Hey all! I was searching how to automatically suspend and resume fabric capacities and leave it running only on week days in order to save money or to scale up wihout exceeding the budget. So, I found this tutorial from Chris and Kevin where they show step by step how to do it. Since it's really useful and I haven't seen this posted here yet I thought of sharing it with you. https://www.youtube.com/watch?v=OHw-RJPW9SQ This is the link for the article they are using: https://richmintzbi.wordpress.com/2023/11/23/microsoft-fabric-capicity-automation/ In the article there is also the code for automatically resizing the capacity which can also be very useful, if you know the time you'll need more compute you can schedule when to scale up and scale down after a period of time.

👋 First things first - welcome to all the new joiners! 🤝 We have a bit of a tradition here for new joiners to introduce themselves just so everyone can say hello and learn about the amazing people (you!) that are regularly joining the community. It also helps me understand more about what you are looking to get out of the community (so I can make it a better experience for you!). 💬 So, don't be shy, we're a friendly bunch... feel free to say hello using the following questions as a guide: 1. Which part of the world are you signing in from? 2. What's your background (current role, industry, ambitions for the future)? 3. What attracted you to this community? 4. What excites you most about Fabric? 5. I'm really interested in learning more about _______ ? 6. Are you looking to get Fabric certified (DP-600)? Thank you for engaging and joining us on this exciting learning journey! 🙏 Will

Hi everyone! Is it possible to query a Fabric Lakehouse or DW using a python script outside Fabric? I have this script that must run only on this specific computer and it will need information from one of the tables in my lakehouse.

Hello everyone. I'd like to start by apologizing for the long text. If anyone manages to get to the end and has an opinion to share or find something that doesn't make any sence, I would be grateful and happy to discuss further 😊 Currently, I am working on building a data pipeline at the company where I work to meet their needs for reporting and analysis. I don't have much practical experience with data engineering yet, this is actually the first complex project I'm working with, and since it primarily involves Fabric, I thought of sharing it with you. What I have defined is that the data - ready for consumption - will be available in silver and gold layers in Fabric lakehouses. Where I am having difficulty deciding is at the initial part of the pipeline where I need to decide between 2 approaches that seem to make sense. This initial part includes an initial data validation by checking constraints and data types, handling SCD cases, and performing retroactive updates of some tables (e.g., stock prices need to be adjusted backwards according to market events). I've attached a diagram that I believe helps visualizing. Basically, there are 3 different data sources, with (i) and (ii) being accessed via API and (iii) consisting mostly of manually filled Excel tables with registration data that will later be used to build dimensions. In option A, an initial ETL process would be done for a PostgreSQL database. The main idea here is to have a very direct way to validate the data through constraints and typing. As for SCD cases and retroactive updates, it seems feasible to perform here as well. In option B, the data would be loaded into a Fabric bronze lakehouse and all this initial stage would need to happen within Fabric. At first glance, this approach seems simpler and less costly because it does not require an additional platform to place the data, nor an additional copy of the data. My concern is whether it will also be simple to perform the tasks I mentioned directly in Fabric.

Hi all! I would love to get your views and experiences with data quality and data validation in data analytics/ viz/ engineering. It would be great to hear your experiences on any of the following: - Is this an area you've worked in before? - If so, what were you doing and - how did you approach it? - Any particular tools or frameworks you used? - What were the outcomes for you and your company/ client? The reason I'm asking is that for me it's probably the most important part of any data strategy, and one that is often neglected. Fabric makes it simpler to implement data quality/ validation rules in nearly every part of the data analytics workflow/ system. If there's any interest, I'll look to create a series of videos on how to think about data quality and validation within Fabric, and how to actually implement measures that grow trust in your analytics systems, visualizations and insights.