Format Run name (or batch id?) in Document Store (to be used in Airflow)

Archive
,
1

Context: We’re using shared Document “Store” on S3 for all our GE runs.
We do run GE in Airflow, and I noticed that finding specific runs in the store is almost impossible.
Here is the issue:

We run Airflow jobs on a schedule, with certain parameters. Sometimes (e.g. when run failed) we have to re-run tasks. So, ideally, it would be great to have some sort of “job/params/hash_or_num” format for the validation results. Unfortunately, right now we only have a target table and some hash. Is it possible to customize run names?

1 Like
2

You can customize this by setting run_name - please use this notebook as a reference: great_expectations/validation_playground.ipynb at develop · great-expectations/great_expectations · GitHub