Can I embed Great Expectations in another Application?

Archive
1

What if I want Great Expectations to run inside of an existing application? For example, if I want it to run within a Dag Runner like Dagster or Airflow, or inside a platform like Ascend, or even inside a proprietary Data Quality Application, is it possible?

Some of the next order follow on questions include the following:

  1. Can we access the expectation library in a store so they can be called from another application?
  2. Can we invoke Great Expectations as a function inside another application?
  3. Can we output validation results inside another application?
2

Yes, many teams run Great Expectations inside pipeline execution/DAG runners.

Here is an example that shows how to run it inside Airflow: https://github.com/superconductive/ge_tutorials/tree/main/ge_dbt_airflow_tutorial It addresses your questions.
ease see the Airflow example.

1 Like
3

We also recently made a guide on Deploying Great Expectations with Airflow which you can find here: https://docs.greatexpectations.io/en/latest/guides/workflows_patterns/deployment_airflow.html