Activity
Mon
Wed
Fri
Sun
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
What is this?
Less
More

Memberships

Learn Microsoft Fabric

14.3k members โ€ข Free

2 contributions to Learn Microsoft Fabric
Loading file with Great Expectations
I'm trying to use the instructions from "End-to-end data validation strategies in Microsoft Fabric" video with my own data. I'm a newbie here and to the notebooks concept. From the video, I'm on the first phase of Schema Validation (and am thankful for the guidance in the video) I have a parameter node where I've defined the parameter like: fileToTest = "Files/_folder_/_fileName_.csv" Second node is %pip install --q great_expectations --- update--- It seems that the pip Install command is causing at least part of my problems. If I comment it out. I can utilize the fileToTest parameter in subsequent cells, but I can't use the dependent functions. Third node is: import great_expectations as gx gxContext = gx.get_context() validator = gxContext.sources.pandas_default.read_csv(fileToTest) I consistently receive a "NameError" for the reference to "fileToTest". I wanted to validate that the fileToTest parameter works so in the same Parameter node I added (started with dragging the file to the notebook) df = spark.read.format("csv").option("header","true").load(fileToTest) display(df) This works, displaying the content of the CSV. What am I missing?
1 like โ€ข Jul '24
Eventually found my issue spark.read can work with a different file path than great_expectations These both work df = spark.read.format("csv").option("header","true").load(fileToTest) validator = gxContext.sources.pandas_default.read_csv("/lakehouse/default/" + fileToTest)
0 likes โ€ข Jul '24
@Will Needham it' wasn't gx. It was the need to prepend/lakehouse/default to the file path compared to the spark.read command, which is what your sample notebook has. I may not be following fully as I'm really wet behind the ears with Fabric
๐Ÿ‘‹ New joiner? Welcome! Start here ๐Ÿ‘‡
Welcome to all new members, here's some links and information to help you get started! ๐—ค๐˜‚๐—ถ๐—ฐ๐—ธ ๐—Ÿ๐—ถ๐—ป๐—ธ๐˜€ ๐˜๐—ผ ๐—ด๐—ฒ๐˜ ๐˜€๐˜๐—ฎ๐—ฟ๐˜๐—ฒ๐—ฑ - For an introduction to this community โ†’ Explore the Welcome Pack - New-ish to Fabric? โ†’ Check out our Fabric Foundation module - Studying for the DP-600? โ†’ Check out the DP-600 Module and the DP-600 category - Studying for the DP-700? โ†’ Check out the DP-700 Module and the DP-700 category - Want to get hands-on? โ†’ Check out Fabric Dojo ๐—›๐—ผ๐˜„ ๐˜๐—ผ ๐—ฒ๐—ป๐—ด๐—ฎ๐—ด๐—ฒ ๐˜„๐—ถ๐˜๐—ต ๐˜๐—ต๐—ฒ ๐—ฐ๐—ผ๐—บ๐—บ๐˜‚๐—ป๐—ถ๐˜๐˜†? - Share your knowledge and experience! Even if you're relatively new to Fabric, or the community, your opinion and experiences are valued here! A great way to earn your first point(s) is to introduce yourself in the thread below ๐Ÿ‘‡๐Ÿ˜€ Thank you for engaging and joining us on this exciting learning journey! ๐Ÿ™ Will
4 likes โ€ข Jul '24
Hi, I'm John from the US. Been working with PowerBI (pro) for a few years and trying to work out what advantages leveraging more of the Fabric stack could bring for us. Specifically interested on taking the 'fact' tables we've created inside our single semantic model and moving them to more accessible ?warehouse tables? and maybe moving away from a single model to rule them all.
1-2 of 2
John Nickell
2
15points to level up
@john-nickell-1494
Data munger and multi-hat wearer at a small family businesss

Active 458d ago
Joined Jul 18, 2024
Powered by