Can I use Great Expectations in an Argo pipeline?

Archive
1

Copying this from a Slack question, since it might be of broader interest:

I’ve seen that Great Expectations has Airflow integration to spawn and manage the compute jobs for the data validation. Pretty neat. However, we’re running all our workflows on Argo (https://argoproj.github.io/). Are there any plans to integrate with that? Or is there any Great Expectations user currently known who has him/herself integrated with Argo whom I can reach out to to exchange learnings?

2

Currently there is no officially supported integration or a how-to guide for implementing Great Expectations data validation in an Argo pipeline.

We believe that the core ideas for such integration can be borrowed from the Great Expectations integration with Airflow: https://docs.greatexpectations.io/en/latest/guides/workflows_patterns/deployment_airflow.html

Please add comments to this post if

  1. You already have done it for themselves and can share any lessons, please add a comment to this post.
  2. You are interested in using this integration
  3. You are interested to develop this integration and contribute it to the project.