Sadly neither option ended up working for me in the end. When trying to use option 1 utilising abfss paths the notebook would hold onto the default workspace I had set when writing it, or if I removed it complain that it could not execute the script without having a default lakehouse set. Option 2 was a no go because although I could parameterise my default lakehouse, I could not set other parameters for use later on in the notebook. I have set the default lakehouse using deployment rules and left the notebooks refactored to use the abfss paths, just encase I ever figure out a way to get it working 😀

@Will Needham Apologies in all of my digging I got mixed up and forgot that your solution was a single approach that involved the notebook working out for itself its local workspace. I was trying to feed this information in using a Fabric pipeline, which in turn made me loose the ability to set other parameters. If you are thinking about making a video on this, that would be amazing, because I cant find anywhere else going into the nuances of this. Here are a couple of things I have come up against, and please feel free to correct anything I have got wrong. 1. The %%configure statement must be in cell 1 of the notebook, otherwise you receive an error "MagicUsageError: The current running Livy session must be restarted for the config changes to take effect. Specify the argument "-f" to force restarting Livy session. Any variables stored in memory will be cleared." This is also where I struggle to implement your solution as I cant use the environment_switcher logic before %%configure statement or within the same cell. 2. You can parameterise the %%configure statement to receive dynamic inputs from a fabric pipeline, but in doing so loose the ability to use other parameters later in the notebook. 3. If you try to work around the issue by not assigning a default lakehouse and only using abfss paths to read and write to other lakehouses in separate workspaces. You will receive an error that the notebook cannot run without a default lakehouse being assigned. In addition to this I have tested leaving a default lakehouse assigned but it also give errors when deployed to another environment.

Hi, I'm Ben, I'm from the UK, I work as a Data Engineer I'm really interested in learning more about CD/CD.