Save Raw Files to Lakehouse from Notebook
Hello All,
I'm new here, new to data science, and new to fabric! I'm trying to consume a Rest API and step one is to save the json response to the bronze layer even before doing any validations. I was hoping to store the result as a json file in the lakehouse but am having difficulty on how to do it.
I was able to convert it to a dataframe but haven't found any resources that show how to save it directly as a json file.
Working DF Example:
from pyspark.sql import Row
def get_token(url, data):
# Make the POST request
response = requests.post(url, json=data)
# Check if the request was successful
if response.status_code == 201:
json_response = response.json()
lakehouse_path = f"Files/auth.json"
data = [Row(**json_response)]
df = spark.createDataFrame(data)
df.write.json(lakehouse_path)
I'm curious how to store the raw files directly (not a dataframe).
I would like to solve this problem as I'm also interested in saving image files which will be fetched from the api at a later date.
Thanks for your time!
0
6 comments
Jacob Kapostins
2
Save Raw Files to Lakehouse from Notebook
Learn Microsoft Fabric
skool.com/microsoft-fabric
Helping passionate analysts, data engineers, data scientists (& more) to advance their careers on the Microsoft Fabric platform.
Leaderboard (30-day)
Powered by