Airflow Integration

Archive
1

I hear lots of questions from users who are integrating Great Expectations into existing pipelines that use Airflow to manage scheduling and execution. As part of a recent meetup, I built a minimal working airflow example that demonstrates a simple deployment pattern of wrapping great expectations execution into a PythonOperator: GitHub - greatexpectationslabs/airflow_meetup_demo

There is also an example of airflow integration in the primary repository here: https://github.com/great-expectations/great_expectations/tree/develop/examples/integrations/airflow

Please feel free to post airflow questions, challenges, successes, tips, or anecdotes here!

3 Likes
2

Here’s what’s on my airflow wishlist:

  • an operator that utilizes a ge DataContext to make loading of suites easy
  • an operator that makes validation easy, that ideally includes notification options
  • great docs for these operators

thoughts?

1 Like
3

Update! Superconductive recently published an updated example of a Great Expectations + Airflow: https://github.com/superconductive/ge_tutorials

1 Like
4

We’ve made it official! Here is a short how-to on Deploying Airflow and GE! https://docs.greatexpectations.io/en/latest/guides/workflows_patterns/deployment_airflow.html