Concurrency Issue with multiple notebooks updating same table

Hello Team,

I am in a need of running 10 to 15 notebooks every hour to sync the data. But the problem that I am facing is, when I scheduled for every one hour, it few notebooks fail with concurrency issue.

And it is not the same notebook that fail every time, different notebook at different time interval.

In all my execution, there is one common sync_log table in which every notebook will SELECT, INSERT and UPDATE records.

Looks like, the concurrency error happens because of this.

But I can't avoid this table as I am using this to maintain the synchronization log.

Can someone help me with this?

@Will Needham

For your reference here is the error:

Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Py4JJavaError, Error value - An error occurred while calling o359.sql.

: io.delta.exceptions.ConcurrentAppendException: Files were added to the root of the table by a concurrent update. Please try the operation again.

4 comments