Hi,
I have Airflow dag runs almost 100 Spark jobs and each spark job has a checkpoint run inside and every checkpoint has action for update data docs that immediately seen from gx UI. When I run the all task from dag sometimes it gives GX validations error about
No such file or directory: ‘/XXXX/gx/uncommitted/data_docs/local_site/validations/XXX__suites/XXX_20260927_27812261/20261105T075844.216073Z/XXX.html’ but ‘XXX’ is not my current validation task name. It gives error for other task. I believe it is concurrency problem because every checkpoint update shared data docs. Is it a problem to write concurrent update to data docs for every checkpoint? Should I update data docs at the end of the process or is there any other way? I do not get what is the problem? (I am new to Great Expectation)
Hey @Kamran, thanks for the question. I can’t say with certainty what’s happening without seeing the stack trace of the error and your config, but it’s very likely a concurrency issue caused by multiple processes writing to the same DataDocs directory. The individual file writes are probably not the issue, but there’s a race condition on the index rewrite that is probably causing this behavior.
Since the UpdateDataDocsAction updates the index without a lock, the best approach is to have a single data docs writer. Remove the UpdateDataDocsAction from your Checkpoint config and move the data docs build to the end of your dag, then have a task call context.build_data_docs() after every checkpoint has completed. You could also have that task run more regularly across the run, either at select places in the dag or on a schedule, if getting close to real-time updates in the UI is valuable. Either way, avoid having two or more task writing Data Docs concurrently, and this intermittent error should go away.
Thanks for your time and quick answer Josh.