Activity
Mon
Wed
Fri
Sun
Jul
Aug
Sep
Oct
Nov
Dec
Jan
Feb
Mar
Apr
May
Jun
What is this?
Less
More

Memberships

Learn Microsoft Fabric

10.9k members โ€ข Free

9 contributions to Learn Microsoft Fabric
Variable delays in data - Lakehouse schema shortcuts
We are currently facing issues with intermittent, and sometimes very long delays in data coming through into shortcut sources (sometimes more than 1 day). we thought we had cracked it by forcing a meta data refresh in the lakehouse through a scheduled notebook by using this command spark.sql('REFRESH /'). However, I found this was not solving the issue today, nor did a manual meta data refresh, which may be because it was stuck or failing to refresh as the spinning icon remained spinning and I had to refresh the page. The schema enabled lakehouse is still in preview so it may just not be ready for production yet.
0 likes โ€ข 28d
@Sambhav R Yes the issue is that they sometimes do not update, or build up a latency. How can caching be disabled?
0 likes โ€ข 26d
@Sambhav R Thanks. We have caching disabled already.
Replication/Mirroring - Anyone got this working?
We have replication from an Azure Sql Server to a Mirrored item. The replication brings in a new parquet file with a few rows as the source db updates (it's a call centre transactional db). It means we have thousands of parquet files being created. It's not possible to run the optimisation command because of permission to edit in the location the mirror uses (the mirror's own Azure blob storage). I believe this fragmented files are putting large strain on the meta data update of the lakehouses which shortcut to this data. I can't be certain but I think this may be causing the delays and lockups of the schema shortcuts downstream that we are seeing. I'm very interested in how others are implementing replication and how it's working with the downstream dataflow.
0 likes โ€ข 26d
@Will Needham Thanks for the reply. We are really hoping this would work as it saves a lot of time in setup and maintenance, this is what we do in our Azure solution which is what we are migrating from into Fabric. The dbs that we are mirroring seem to all have entire tables brought in multiple times as well (> 20 times per day) which suggests to me that fabric mirroring is not working properly, despite being out of preview. I've got an open ticket with Microsoft regarding the inconsistent syncing of the schema shortcuts to the mirror db. I'll see if they can give any insight into the mirroring feature and how shortcuts deal with the abundance of .parquets.
Fabric reports running in apps alongside legacy BI Pro reports
Hi all, We are trying to get a report from fabric integrated into an app that is non-fabric premium workspace to be able to keep it alongside our legacy reports. We have published a report using BI desktop and the semantic model within a fabric workspace. The dev team can see the app but I'm struggling to get permissions to work for anyone without fabric workspace access to view the report. I have given users access to the semantic model but this is not enough apparently.
Starter Pool - Start-up Time
Hi All, A StarterPool is supposed to spin up in 10 seconds or so. I'm trying to get this to work but I still have start up times of 3-4 minutes in my notebooks, whether I use Starter Pool or other custom pools. Is there some way of forcing the "always-on" pool?
0 likes โ€ข Apr 11
We have it working now and my understanding of how it works has increased. The final setting which got it working was this:
0 likes โ€ข Apr 11
Starter pools are machines with a standard config and no additional libraries, they are already running and ready for consumers to take them over, so naturally, because they have already been spun up, you cannot install additional libraries or set certain properties on them.
What are you working on this week?? + FabCon
Hey everyone, it's been a busy few months working on the DP-700 content! But it's mostly finished now - best of luck to everyone taking the exam in the coming days and weeks! What are you working on this week? Also... I'll be posting more in this community again going forward ๐Ÿ˜€ ESPECIALLY as next week is FabCon - I was speaking to some Microsoft people last night and there are LOTS of interesting announcements to be made! Some questions for you ahead of FabCon: - Is anyone going to FabCon this year? I'm not going this year (maybe EU FabCon later in the year!) - What feature are you waiting for/ excited for? Let's hear it! Next week, I plan to post quite a bit with all the updates! I also have plans to do some collabs with Microsoft PMs for some interesting new features for the channel, so watch this space ๐Ÿ˜€
2 likes โ€ข Mar 25
Our team is booked in for FabCon EU!
1 like โ€ข Mar 25
This week we are working on a pilot project to get a near-live dashboard from call centre data. The data is replicated into Azure, mirrored into fabric, shortcutted to a lakehouse; we will then attempt to transform the data into a star schema using the following methods for comparison: a view in SQL endpoint , Dataflow Gen2 and a Notebook in a pipeline. If anyone has any insight into the merits and pitfalls of these approaches It'll be welcomed.
1-9 of 9
Renรฉ Tachon
2
8points to level up
@rene-tachon-4307
After 15 years in engineering I moved to data with aspirations of becoming a Data Scientist. I am now working more broadly in the data space.

Active 26d ago
Joined Sep 18, 2024
powered by