Copying this from a Slack question, since it might be of broader interest:
I’ve seen that Great Expectations has Airflow integration to spawn and manage the compute jobs for the data validation. Pretty neat. However, we’re running all our workflows on Argo (https://argoproj.github.io/). Are there any plans to integrate with that? Or is there any Great Expectations user currently known who has him/herself integrated with Argo whom I can reach out to to exchange learnings?
Currently there is no officially supported integration or a how-to guide for implementing Great Expectations data validation in an Argo pipeline.
We believe that the core ideas for such integration can be borrowed from the Great Expectations integration with Airflow: https://docs.greatexpectations.io/en/latest/guides/workflows_patterns/deployment_airflow.html
Please add comments to this post if
- You already have done it for themselves and can share any lessons, please add a comment to this post.
- You are interested in using this integration
- You are interested to develop this integration and contribute it to the project.