I get it, it's an attractive proposition. Type any technical question into a chat window and get an instant response. Unfortunately (at the moment), it's not quite as simple as that. I think we all know that ChatGPT & other large language models (LLMs) can hallucinate, i.e. confidently giving you answers that: - are wrong - are misleading - were maybe right 6 months ago, but now the answer is irrelevant/ not accurate. With Fabric, they are a few factors that increase the likelihood of hallucinations, that you need to be very aware of: - Fabric is fast moving - things change weekly, monthly. Therefore a feature/ method/ piece of documentation that was used in the last LLM training run 6 months ago, might no longer be relevant, or new features have superseded previous approaches. - Fabric is the evolution of previous Microsoft data products. This is good in some ways, but catastrophic for LLMs (and learners relying on LLMs). There is vastly more training data out on the internet for Azure Data Factory, for example, than Fabric Data Factory. Or Azure Synapse Data Engineering over Fabric Data Engineering. And yes there are similarities for how the old tools work vs the new tools, but you need to be super careful that the LLM generates a response for FABRIC Data Pipelines, rather than Azure Data Factory pipelines, for example. Or generates Fabric Data Warehouse compliant T-SQL code, rather than Azure SQL code. This is very difficult, unless you have knowledge of how both products work (which most learners/ beginners don't!). I'm not saying don't use LLMs for studying, just that you need to be super careful. I can think of two use cases that are lower risk, using LLM+Fabric for Spark syntax & KQL syntax generation. That's because Spark and KQL are very mature ecosystems, with lots of training data on the internet, and their syntax won't change too much over the months and years. Fabric Data Warehouse T-SQL code generation is more tricky/ risky because the way the Fabric Data Warehouse works is quite different to a conventional SQL Server (which is what most of the training data will be based on).

Hi All, A know issue is the fact that when I ingest data into a lakehouse, there can be (nah will be :)) a delay before that data is actually available in the SQL analytics endpoint. Up until now I have been trying to tackle this by adding 5 or even 10-minute delays between lakehouse ingestion steps and addressing the SQL endpoint to do joins to dim tables, ETL etc. This works most the time but of course causes delays in running pipelines (and it is not 100% proof). I came across the below MS article, has anyone tried this yet and is it really as simple as adding a script activity in the pipeline along the lines of: "SELECT TOP(1) 1 FROM [lakehouse].[dbo].[tablename]" Or should i read all rows, like SELECT 1 and not top (1) ? https://learn.microsoft.com/en-gb/fabric/known-issues/known-issue-1092-delayed-data-availability-analytics-endpoint-pipeline?wt.mc_id=fabric_inproduct_knownissues Cheers Hans

Hey everyone, just a quick one - Microsoft have released the official Practice Questions for the DP-700: https://learn.microsoft.com/en-us/credentials/certifications/fabric-data-engineer-associate/?practice-assessment-type=certification#certification-practice-for-the-exam If you're studying for the DP-700 exam, I definitely recommend testing your knowledge using this official resource!

One of the reoccurring issues that people face in Fabric (and post about here!) is the delay it can take for the T-SQL endpoint of a Lakehouse to update, when you make changes to Lakehouse data and tables (via the Lakehouse/ Spark endpoint). This is most problematic when you have a multi-step ETL/ELT process that first updates some Lakehouse table, then reads that data (via the T-SQL endpoint), using a T-SQL script in a Warehouse. Well, there is a workaround to force the update of the T-SQL endpoint, using Semantic Link. This workaround was provided initially by the Microsoft product manager for the Lakehouse T-SQL endpoint. Some links on this: - Original LinkedIn post by Mark (Microsoft) - Obvience video walking through how to implement it. Hopefully they improve the native functionality in the future, so that forced updates are not required, but in the short-term, this should help a lot of people.

Hi all, in order to connect Fabric to an API, I need to use a password and this password is stored as a variable in a pipeline. This works, but leaves the password visible for anyone having access to this pipeline. Is there some way around this?