Updating data docs

Hey @erman I replied on your message on gx-community-support slack, mentioning the same message here:

So we no longer use Config-driven methods so first, we recommend that you fully upgrade to the Fluent Data Source (FDS) method that you are implementing partially. For instance, to call Checkpoint, you should use context.add_or_update_checkpoint method. If you need visual assistance, here’s a flowchart: Miro | Online Whiteboard for Visual Collaboration

Now as to your general issue, this is pretty common, particularly for folks with a huge amount of historical validations, or folks running massive validations regularly. If you’re using a BuildDataDocs action (which I can’t tell in your case as I’m not familiar with deprecated config method), every one of those html files will be re-built, which can eventually take quite a long time – swapping this for an UpdateDataDocs action will resolve that issue. When you run the FDS method, I believe it should default to UpdateDataDocs action in the checkpoint yml file that gets generated once the checkpoint is run (the yml will be under your checkpoint folder). If not, change to this action. UpdateDataDocs will only build docs for new validations, rather than rebuilding all docs. Update Data Docs after Validating a Checkpoint | Great Expectations

Alternatively, you could remove both data docs actions from your workflow, and run context.build_data_docs() off cadence outside of that workflow to isolate and minimize that process.