Data pipeline - Upsert table action · Learn Microsoft Fabric

Aug 12 (edited) • Technical

Data pipeline - Upsert table action

I have on-premise Microsoft SQL servers connected to my tennant through a data-gateway. I have a pipeline in place that utilises a watermark strategy, i am attempting to ingest partial tables using a select statement, when i attempt to utilise the 'upsert' table action on my destination Lakehouse, it results in the below error:

"ErrorCode=FailedToUpsertDataIntoDeltaTable,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Hit an error when upsert data to table in Lakehouse. Error message: Could not load file or assembly 'System.Linq.Async, Version=6.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263' or one of its dependencies. The system cannot find the file specified.,Source=Microsoft.DataTransfer.Connectors.LakehouseTableConnector,''Type=System.IO.FileNotFoundException,Message=Could not load file or assembly 'System.Linq.Async, Version=6.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263' or one of its dependencies. The system cannot find the file specified.,Source=Microsoft.DataTransfer.Connectors.LakehouseTableConnector,'"

I have created multiple pipelines in multiple workspaces, stripping things back to a simple 'copy data' activity, and i always recieve this when i try to make use of upsert, does anyone else recieve this?

When using append, It seemingly results in duplication / old outdated stale rows (and a new created with the change made to a particular column within the row)

I am aware of overwrite being an option also but i do not wish to go down this route.

Im a bit stuck currently on being able to ingest data cleanly and without duplication due to this, has anybody seen this before, or have any suggestions as a workaround? The error is very indicative of a platform issue, something compelltey out of my control...

5 comments