@Lori Keller I can see your point. We have all in apps, stayed also in the MS corner of the ring until now. Have need for Sparks driven by notebooks and run by a pipeline for efficient processing of a new data stream coming in. And the data is complex so switched to do it in Python this time.

@Will Needham Already live? Since yesterday I have production code failing which worked before, even simplifying to a regular one (no haversine calculations, no views or old statistics etc.) keeps failing: INSERT INTO #BSPTable ( [SQNR], [DAIS], [TDNR], [TRNN], [ORAC], [DSTC], [CouponPrice], [OracLat_Raw], [OracLon_Raw], [DstcLat_Raw], [DstcLon_Raw] ) SELECT BKI63.[SQNR], BKI63.[DAIS], BKI63.[TDNR], BKI63.[TRNN], BKI63.[ORAC], BKI63.[DSTC], 0, APOrac.Latitude, APOrac.Longitude, APDstc.Latitude, APDstc.Longitude FROM [BSP].ItineraryDataSegment AS BKI63 LEFT JOIN [BSP].OracFlightsAirports AS APOrac ON BKI63.[ORAC] = APOrac.IATA LEFT JOIN [BSP].[DstcFlightsAirports] AS APDstc ON BKI63.[DSTC] = APDstc.IATA -- WHERE clause might be applied here if needed, or later WHERE BKI63.[TRNN] IN ('000010','000035','000068','000114','000140','000141') -- Example filter ; Msg 15816, Level 16, State 7, Line 115The query references an object that is not supported in distributed processing mode. Aware that temp tables can become a culprit when using too much complexity in the following code, but this stuff was working before, that is the annoying part.

Be aware that Fabric has multiple URL's to enter, so port 1433 needs to be open on multiple of them, you notice this when running SSMS / ADS on prem sitting behind a corporate firewall instead of working from home. Microsoft has a support page for this.

if it is unfinished as current deployment pipelines (DP), I say keep it and cover first your base before moving on, at least take care that automatic binding isn't auto and could be FULLY parameter driven. Lots of folks using DFG2 would then happily also test this to, like me!

@Will Needham Thx for asking! Having different data streams from separate Service Bus queues for the Dev / Sta / Prd, I need to be able to make sure that pipelines and semantic models have bindings to the proper data sources. Likewise ADF I expected parameters to handle this. I am aware you can go the proxy route or hack beneath the given current gui, but it is ugly and it feels like the auto binding behavior is more a temporary bandage than a given solution. So don't get me wrong I am not bad mouthing here, else I would have waited for a year longer with already moving from ADF to Fabric. Nevertheless now it's just a very simple Bronze/Silver/Gold implementation. In short we are first going to test with hardcoding the source and there use variables which are tied to the parameters we do have available in the dev pipelines and take it from there until we have proven solutions for all our adventures on the road to implement the architecture we have in mind for our needs and wishes ;-)