Activity
Mon
Wed
Fri
Sun
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
What is this?
Less
More

Memberships

Learn Microsoft Fabric

14.3k members โ€ข Free

5 contributions to Learn Microsoft Fabric
Save Raw Files to Lakehouse from Notebook
Hello All, I'm new here, new to data science, and new to fabric! I'm trying to consume a Rest API and step one is to save the json response to the bronze layer even before doing any validations. I was hoping to store the result as a json file in the lakehouse but am having difficulty on how to do it. I was able to convert it to a dataframe but haven't found any resources that show how to save it directly as a json file. Working DF Example: from pyspark.sql import Row def get_token(url, data): # Make the POST request response = requests.post(url, json=data) # Check if the request was successful if response.status_code == 201: json_response = response.json() lakehouse_path = f"Files/auth.json" data = [Row(**json_response)] df = spark.createDataFrame(data) df.write.json(lakehouse_path) I'm curious how to store the raw files directly (not a dataframe). I would like to solve this problem as I'm also interested in saving image files which will be fetched from the api at a later date. Thanks for your time!
0 likes โ€ข Jun '24
@Will Needham I'm having trouble copying raw files between two lakehouses. I'm struggling to read/write to the "files" section of the lakehouse if its not set as the default. I attached a screenshot showcasing my difficulty. Sorry for the direct tag, I was hoping to catch you in your monthly Q/A session.
0 likes โ€ข Jun '24
@Will Needham thanks for responding. if you check the screenshot i believe that's what I tried in my 'silver_path2' attempt. I should note that your suggestion works as expected when using the abfss path on tables with spark libraries, however for raw files I'm failing. I'm guessing its something to do with the 'os' or 'Path' libraries I'm using
IT'S FRIDAY!! So share your Fabric Wins...
Learning Fabric is a marathon, not a sprint. So let's celebrate the small wins we make along the journey. Share any recent wins you've had with Fabric in the comments below! Perhaps this week you: - successfully shortcuted into Amazon S3? - built a Snowflake mirrored database? - learnt some new data modeling techniques? - build your first eventstream to process real-time data? LET'S HEAR ABOUT IT!
IT'S FRIDAY!! So share your Fabric Wins...
3 likes โ€ข Jun '24
-Created my first notebook to fetch data from an API -Learned about medallion architecture, created bronze and silver lake houses -Stored raw api responses as json files in bronze lakehouse -Transformed and validated responses with great expectations and stored them as tables in the silver lakehouse -Great expectations results are written to a "data quality" lake house so I can analyze success and failures -Created data pipeline to orchestrate calling my notebooks
0 likes โ€ข Jun '24
@Will Needham Thank you for the content and teaching me about it!
The countdown to ending Fabric Trials ๐Ÿ˜ฑ โฐ Are you prepared?
My 60-day free trial is coming to an end... after 383 days (thanks Microsoft) ๐Ÿ˜‚ Yours probably is too. The question is, are you prepared? What are your options? 1. You can purchase a F-SKU capacity from the Azure Portal, and reassign your existing workspaces to that new capacity. Or, 2. If you're still not ready to buy an F-SKU yet, the documentation states you can create a new trial capacity with a new user, and reassign your workspaces to that new trial capacity. 3. Any others? Let me know! Important things to note: - If you don't upgrade to a paid Fabric capacity before the end of the trial period, non-Power BI Fabric items are removed according to the retention policy upon removal (I believe this is 7 days soft delete, but double check, and also don't let it get to this!)
Poll
43 members have voted
The countdown to ending Fabric Trials ๐Ÿ˜ฑ โฐ Are you prepared?
0 likes โ€ข Jun '24
@Darin Spence I would be interested in this as well. I would love to have a "serverless" capacity so we can pay for what we use. My capacities will be sitting idle for the majority of the week.
Read/Write data to fabric lakehouse from a Notebook in VS Code?
When authoring a notebook from the fabric web interface I see an option to open the notebook in VS code. This opens the notebook locally on my PC. The issue I'm dealing with is that is not able to authenticate with the workspace / lake house from my local notebook. I assume there is a something we need to do to configure spark authentication, however when browsing the lakehouse in fabric i'm only able to find the "SQL Connection String" but no other credentials. I assume this is a solve problem since the main purpose of these notebooks is to read/write data, however I can't find any documentation on it. Example: flat_df.write.format("delta").mode("overwrite").save("Tables/auth_result") The above works from the online fabric notebook, but fails with the following error when opening the notebook with VS Code and trying to execute the same line. ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it Any help would be greatly appreciated!
0 likes โ€ข Jun '24
Thank you @Paul McGuigan !
๐Ÿ‘‹ New joiner? Welcome! Start here ๐Ÿ‘‡
Welcome to all new members, here's some links and information to help you get started! ๐—ค๐˜‚๐—ถ๐—ฐ๐—ธ ๐—Ÿ๐—ถ๐—ป๐—ธ๐˜€ ๐˜๐—ผ ๐—ด๐—ฒ๐˜ ๐˜€๐˜๐—ฎ๐—ฟ๐˜๐—ฒ๐—ฑ - For an introduction to this community โ†’ Explore the Welcome Pack - New-ish to Fabric? โ†’ Check out our Fabric Foundation module - Studying for the DP-600? โ†’ Check out the DP-600 Module and the DP-600 category - Studying for the DP-700? โ†’ Check out the DP-700 Module and the DP-700 category - Want to get hands-on? โ†’ Check out Fabric Dojo ๐—›๐—ผ๐˜„ ๐˜๐—ผ ๐—ฒ๐—ป๐—ด๐—ฎ๐—ด๐—ฒ ๐˜„๐—ถ๐˜๐—ต ๐˜๐—ต๐—ฒ ๐—ฐ๐—ผ๐—บ๐—บ๐˜‚๐—ป๐—ถ๐˜๐˜†? - Share your knowledge and experience! Even if you're relatively new to Fabric, or the community, your opinion and experiences are valued here! A great way to earn your first point(s) is to introduce yourself in the thread below ๐Ÿ‘‡๐Ÿ˜€ Thank you for engaging and joining us on this exciting learning journey! ๐Ÿ™ Will
3 likes โ€ข Jun '24
Hi everyone! I'm a software engineer turned cto. I love to learn about technologies and apply my learnings in my business + every day life. I'm new to data science and fabric. I'm eager to learn all about it!
1-5 of 5
Jacob Kapostins
2
11points to level up
@jacob-kapostins-4764
Founder

Active 429d ago
Joined May 29, 2024
Powered by