Activity
Mon
Wed
Fri
Sun
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
What is this?
Less
More
Memberships
Learn Microsoft Fabric
14.3k members ā¢ Free
2 contributions to Learn Microsoft Fabric
Mar '24 ā¢Ā
Question For Anyone Using Mirroring (private preview)
I was wondering has anyone here used mirroring in private preview? I was wondering to what extent the storage location structure (folder structure or partitioning) of the real-time mirror is configurable, for example, to use for mirroring a multi-tenant database to a different location (top-level folder) per tenant, such that the data for each tenant can already be isolated & partitioned when it comes in in real-time?
2 likes ā¢ Mar '24
the thing that worries me is the following remark in the docs: "If there are no updates in the source tables, the replicator engine will start to back off with an exponentially increasing duration, up to an hour. The replicator engine will automatically resume regular polling after updated data is detected." So the data is only near real-time if regular updates are done in the source. If it's an application database that is mirrored and the application is not used for a while (during the night for example), then worst case scenario the first change capture will only be processed an hour later :-(
0 likes ā¢ Mar '24
@Julio Ochoa what do you mean 'if there is no data'?? The docs say, if there are no updates in the source table (the situation that the cdc tables contain no new records), then if that is the case for long enough, the replicator engine will back off (for some duration that exp. increases and is max 1 hour), and it will resume polling only after updated data is detected again (which worst case scenario might take an hour for it to detect new records in the cdc tables, right?). Or am I misunderstanding these sentences in the docs, that could also be the case. My preliminary conclusion is that the mirroring is only near real-time all the time when the source is also updated regularly, of course depending on the specifics. And is that mechanism table based or database based (i.e. does stop polling each specific table when that table does not receive any updates, or is it only a single table in the database that needs to be updated regularly for the regular polling of all tables not to stop temporarily)? If it's database based it could easily be manipulated by regularly updating a dummy table in the source application that is used in the mirroring, that would keep regular polling for all the tables active....
š
Feb '24 ā¢Ā
Anyone using real-time data in Fabric? Listen up!
Today Microsoft announced the Eventhouse... you can read all about it here. Is this something you'll be using? Let me know and I can create some videos on it! Key Features of Eventhouses: 1. Workspace of Databases: - An Eventhouse is essentially a shared workspace for databases, facilitating efficient management across specific projects. - Simultaneous management of multiple databases optimizes performance and reduces costs. - Unified monitoring and management are available across all databases and on a per-database basis. 2. Tailored to Time-Based, Streaming Events: - Specifically designed for time-based, streaming events supporting various data formats. - Ingest data from diverse sources and pipelines like Eventstream, SDKs, Kafka, Logstash, and more. - Automatic indexing and partitioning based on ingestion time. 3. Minimum Consumption: - Optimizes cost by allowing the suspension of the service when not in use. - Minimum consumption ensures the service is always available at a selected minimum level, minimizing latency upon reactivation. - Applicable for time-sensitive systems intolerant to latency.
0 likes ā¢ Mar '24
Definitely interested as my company is looking for ways to RT analyze IoT data as well as integrating it with the master data stored in sql. I guess it will be possible to reference / query the eventhouse and the warehouse from a single notebook (for example)?
1-2 of 2
Active 481d ago
Joined Mar 15, 2024
Powered by