Hi Krishan - Good project to start with and nice documentation of your work and goals! I'm thinking about documenting one of my projects as. However, I'm afraid that I have to black out too much stuff, because I only have complete projects done for clients.

Just my 2 cents: Is that the right use case for Fabric? Because it has a database engine to access your data, it is not an operational database system. Fabric is an analytics platform that is able to ingest and process data, analyse that data, train ML on it, and do reporting on that data. I'd go with a stand alone power app and use it's datastore as a source for a Fabric workflow. My opinion is not final, I'm open for better suggestion. I even have a similar requirement in my backlog. So happy to learn from this thread 😆.

In my experience it depends on source and connection options to that source. If your just ingesting from structured data sources, Power Query might be a perfect solution. PySpark comes in when you have more requirements. E.g.: - moving ingested source data files to a specific location - checking and recording the SHA256 hashes of source files - complex data sanitizing - data validation - etc.

My biggest pain point is the lack of support for certain items and scenario's in the deployment pipeline. Second comes the random fallout of functionality. E.g. failure of creating a spark session for a notebook execution. When posting in the Fabric Community yields answers like "come back tomorrow and try again". Besides that, I miss a simple scheduling mechanism to turn on and off a capacity. I'm working for a small company with external managed Azure resources. To turn the capacity on and off, I have to issue a support ticket. So we stick to an "always on" scenario. To reduce costs, we picked a small capacity (F4). That's doable for a single user - single usecase situation, but to small for an environment with more users. E.g. F4 implies 4 concurrent Spark Nodes. The smallest setting on a Spark environment is 1-3 nodes. Effectively that means that there can be one Spark session running at any time. Since every notebook uses it's own spark session, we have to turn off every session after a run. Otherwise you're blocking a co-worker (or yourself when switching to another notebook). If we can easily reduce capacities to only be active at office hours (say 60 hours per week), we can go for a F16 at almost the same monthly costs. This may seem to be a minor issue, but for us, it's big.

You can get the lakehouse id from the url of the lakehouse. Just open the target lakehouse en copy the last guid on the url.