# \#airflow

**URL:** https://discourse.greatexpectations.io/tag/airflow/1.md

[Latest](https://discourse.greatexpectations.io/latest.md) · [Categories](https://discourse.greatexpectations.io/categories.md) · [Tags](https://discourse.greatexpectations.io/tags.md)

---

## [Unable to Apply Multiple Conditions in row\_condition with SQLAlchemy (Snowflake) – acryl-datahub-gx-plugin 1.3.1](https://discourse.greatexpectations.io/t/unable-to-apply-multiple-conditions-in-row-condition-with-sqlalchemy-snowflake-acryl-datahub-gx-plugin-1-3-1/2333)

<div class="topic-metadata">

**Author:** [@prasath](https://discourse.greatexpectations.io/u/prasath)\
**Replies:** 0\
**Last updated:** [February 13, 2026, 2:23pm UTC](https://discourse.greatexpectations.io/t/unable-to-apply-multiple-conditions-in-row-condition-with-sqlalchemy-snowflake-acryl-datahub-gx-plugin-1-3-1/2333 "2026-02-13T14:23:45Z")

</div>

Hi, I’m using acryl-datahub-gx-plugin==1.3.1 for Great Expectations validation and pushing the results/metadata to DataHub. As per the documentation here, I’m defining the row\_condition like this (using SQLAlchemy with…

---

## [Config variable substitution in great\_expectations.yml](https://discourse.greatexpectations.io/t/config-variable-substitution-in-great-expectations-yml/2272)

<div class="topic-metadata">

**Author:** [@JoaoLarrosa](https://discourse.greatexpectations.io/u/JoaoLarrosa)\
**Replies:** 0\
**Last updated:** [July 30, 2025, 8:27pm UTC](https://discourse.greatexpectations.io/t/config-variable-substitution-in-great-expectations-yml/2272 "2025-07-30T20:27:49Z")

</div>

Hi, guys! How you doing? :smiley: I’ve been studying GX to implement it in the company I work. We use Snowflake as our datalake. I’ve been facing trouble to replace variables declared in config\_variables.yml in great\_ex…

---

## [GE run fails on AWS Airflow with File System \[Errno 30\] Read-only file system: '/usr/local/airflow/dags/data-quality/gx'; 3080](https://discourse.greatexpectations.io/t/ge-run-fails-on-aws-airflow-with-file-system-errno-30-read-only-file-system-usr-local-airflow-dags-data-quality-gx-3080/1704)

<div class="topic-metadata">

**Author:** [@zfromsyd](https://discourse.greatexpectations.io/u/zfromsyd)\
**Replies:** 6\
**Last updated:** [July 16, 2025, 5:20pm UTC](https://discourse.greatexpectations.io/t/ge-run-fails-on-aws-airflow-with-file-system-errno-30-read-only-file-system-usr-local-airflow-dags-data-quality-gx-3080/1704 "2025-07-16T17:20:58Z")

</div>

Hi, We are upgrading Great Expectations to 0.18.10. We are currently experiencing a blocker issue from one of the packages that throws a read-only file system error. great-expectations version 0.18.10 airflow-provider…

---

## [Unable to configure the GE checks to validate parquet files on s3](https://discourse.greatexpectations.io/t/unable-to-configure-the-ge-checks-to-validate-parquet-files-on-s3/2141)

<div class="topic-metadata">

**Author:** [@sourabh.somvanshi0](https://discourse.greatexpectations.io/u/sourabh.somvanshi0)\
**Replies:** 1\
**Last updated:** [March 31, 2025, 3:44pm UTC](https://discourse.greatexpectations.io/t/unable-to-configure-the-ge-checks-to-validate-parquet-files-on-s3/2141 "2025-03-31T15:44:15Z")

</div>

I am currently accessing parquet files from aws s3 and using AWS MWAA (Airflow) to validate input files. I am using the dataframe approach however I want to access the files directly from s3 without reading into data fra…

---

## [GX Operator cannot locate my aws credentials](https://discourse.greatexpectations.io/t/gx-operator-cannot-locate-my-aws-credentials/1898)

<div class="topic-metadata">

**Author:** [@devan.domer](https://discourse.greatexpectations.io/u/devan.domer)\
**Replies:** 0\
**Last updated:** [September 12, 2024, 1:20pm UTC](https://discourse.greatexpectations.io/t/gx-operator-cannot-locate-my-aws-credentials/1898 "2024-09-12T13:20:12Z")

</div>

I am using an InferredAssetS3DataConnector to validate data in S3 using GX version 0.18.19 and was able to get a successful manual checkpoint run. I’m now trying to set up an airflow operator to run these validations dai…

---

## [How to Install Great Expectations on Amazon MWAA](https://discourse.greatexpectations.io/t/how-to-install-great-expectations-on-amazon-mwaa/1356)

<div class="topic-metadata">

**Author:** [@SIDDHI](https://discourse.greatexpectations.io/u/SIDDHI)\
**Replies:** 1\
**Last updated:** [August 31, 2023, 4:26pm UTC](https://discourse.greatexpectations.io/t/how-to-install-great-expectations-on-amazon-mwaa/1356 "2023-08-31T16:26:35Z")

</div>

Can you guide us on the right way to install great expectations on Amazon MWAA.

---

## [Airflow running great expectations in DEV and PROD env](https://discourse.greatexpectations.io/t/airflow-running-great-expectations-in-dev-and-prod-env/1339)

<div class="topic-metadata">

**Author:** [@lionel](https://discourse.greatexpectations.io/u/lionel)\
**Replies:** 2\
**Last updated:** [August 24, 2023, 10:52am UTC](https://discourse.greatexpectations.io/t/airflow-running-great-expectations-in-dev-and-prod-env/1339 "2023-08-24T10:52:21Z")

</div>

Checkpoint Run Result \`\`\` in DEV are faster when compared to PROD. it takes around 10-20 seconds in DEV while PROD takes more than 5 minutes. Which makes my process slow which is not what we want in our process of data …

---

## [Version 0.16.10 vs 0.17.11 permission denied](https://discourse.greatexpectations.io/t/version-0-16-10-vs-0-17-11-permission-denied/1337)

<div class="topic-metadata">

**Author:** [@lionel](https://discourse.greatexpectations.io/u/lionel)\
**Replies:** 1\
**Last updated:** [August 23, 2023, 3:32pm UTC](https://discourse.greatexpectations.io/t/version-0-16-10-vs-0-17-11-permission-denied/1337 "2023-08-23T15:32:19Z")

</div>

Hello , I was using version 0.16.10 and i was not facing this error when i upgraded to 0.17.11 i get this error great\_expectations.exceptions.exceptions.GitIgnoreScaffoldingError: Could not create .gitignore in becaus…

---

## [How to pass an Airflow DAG's parameter to GE using OS environment variable](https://discourse.greatexpectations.io/t/how-to-pass-an-airflow-dags-parameter-to-ge-using-os-environment-variable/1344)

<div class="topic-metadata">

**Author:** [@Alex](https://discourse.greatexpectations.io/u/Alex)\
**Replies:** 0\
**Last updated:** [August 23, 2023, 7:51am UTC](https://discourse.greatexpectations.io/t/how-to-pass-an-airflow-dags-parameter-to-ge-using-os-environment-variable/1344 "2023-08-23T07:51:03Z")

</div>

I have Airflow DAGs set up to run Great Expectations’ checkpoints.yml alongside corresponding expectations.json files. These DAGs work well for full Data Quality tests. Now, I’m in need of a DAG that can be triggered wi…

---

## [Can a single expectation suite contain multiple tables with different expectations for different tables](https://discourse.greatexpectations.io/t/can-a-single-expectation-suite-contain-multiple-tables-with-different-expectations-for-different-tables/1301)

<div class="topic-metadata">

**Author:** [@lionel](https://discourse.greatexpectations.io/u/lionel)\
**Replies:** 1\
**Last updated:** [July 26, 2023, 2:59pm UTC](https://discourse.greatexpectations.io/t/can-a-single-expectation-suite-contain-multiple-tables-with-different-expectations-for-different-tables/1301 "2023-07-26T14:59:07Z")

</div>

I want to run GE against snowflake tables in a DB can i have one suite that specifies a expectation for a particular table. thanks

---

## [Processing large tables with AirFlow](https://discourse.greatexpectations.io/t/processing-large-tables-with-airflow/1067)

<div class="topic-metadata">

**Author:** [@Ruslan](https://discourse.greatexpectations.io/u/Ruslan)\
**Replies:** 0\
**Last updated:** [June 30, 2022, 3:22pm UTC](https://discourse.greatexpectations.io/t/processing-large-tables-with-airflow/1067 "2022-06-30T15:22:52Z")

</div>

Hi all! We want to create UI for great expectation suite enabled in AirFlow. But what we see is that we cannot avoid both creating temporal tables, preparing dataframe and reading all the dataset. That is critical for…

---

## [Increased execution time for expectations in Airflow](https://discourse.greatexpectations.io/t/increased-execution-time-for-expectations-in-airflow/1035)

<div class="topic-metadata">

**Author:** [@vhernandezs](https://discourse.greatexpectations.io/u/vhernandezs)\
**Replies:** 1\
**Last updated:** [May 10, 2022, 12:45pm UTC](https://discourse.greatexpectations.io/t/increased-execution-time-for-expectations-in-airflow/1035 "2022-05-10T12:45:08Z")

</div>

Dear Team. I am currently facing an issue while working with GreatExpectations in Airflow. In my current setup, I define several Airflow tasks to run GreatExpectations Checkpoints (version 0.13.37) to validate my data, …

---

## [Using Great expectations in Production environment- Combining multiple data docs, Triggering warning for unfit data](https://discourse.greatexpectations.io/t/using-great-expectations-in-production-environment-combining-multiple-data-docs-triggering-warning-for-unfit-data/965)

<div class="topic-metadata">

**Author:** [@Rutuja](https://discourse.greatexpectations.io/u/Rutuja)\
**Replies:** 0\
**Last updated:** [January 8, 2022, 5:56am UTC](https://discourse.greatexpectations.io/t/using-great-expectations-in-production-environment-combining-multiple-data-docs-triggering-warning-for-unfit-data/965 "2022-01-08T05:56:35Z")

</div>

Hi, I was working on integrating great\_expectations into the production environment where the raw data comes in daily and has to pass through the expectation suites and validations. Is there a way in which all the datado…

---

## [How to set up GE in Google's Cloud Composer environment?](https://discourse.greatexpectations.io/t/how-to-set-up-ge-in-googles-cloud-composer-environment/954)

<div class="topic-metadata">

**Author:** [@Rafal](https://discourse.greatexpectations.io/u/Rafal)\
**Replies:** 0\
**Last updated:** [December 16, 2021, 12:14pm UTC](https://discourse.greatexpectations.io/t/how-to-set-up-ge-in-googles-cloud-composer-environment/954 "2021-12-16T12:14:15Z")

</div>

Hi all I have spent an entire day now on trying to figure out how to install GE in the Google Cloud so that I could run my GE data validation tasks via DAGs in the Cloud Composer. I found that recommended way of doing i…

---

## [Deploying Great Expectations with Google Cloud Composer (Hosted Airflow)](https://discourse.greatexpectations.io/t/deploying-great-expectations-with-google-cloud-composer-hosted-airflow/379)

<div class="topic-metadata">

**Author:** [@anthony](https://discourse.greatexpectations.io/u/anthony)\
**Replies:** 3\
**Last updated:** [July 30, 2021, 6:59pm UTC](https://discourse.greatexpectations.io/t/deploying-great-expectations-with-google-cloud-composer-hosted-airflow/379 "2021-07-30T18:59:56Z")

</div>

This article is for comments to: Deploying Great Expectations with Google Cloud Composer (Hosted Airflow) — great\_expectations documentation Please comment +1 if this How-to Guide is important to you.

---

## [Trigger a Slack notification with the greatExpectationOperator](https://discourse.greatexpectations.io/t/trigger-a-slack-notification-with-the-greatexpectationoperator/777)

<div class="topic-metadata">

**Author:** [@shaharp](https://discourse.greatexpectations.io/u/shaharp)\
**Replies:** 1\
**Last updated:** [June 8, 2021, 10:43am UTC](https://discourse.greatexpectations.io/t/trigger-a-slack-notification-with-the-greatexpectationoperator/777 "2021-06-08T10:43:25Z")

</div>

Hi, It is possible to trigger a Slack notification with the greatExpectationOperator in Airflow? Thanks.

---

## [Create customize expectation and use it in Airflow DAG](https://discourse.greatexpectations.io/t/create-customize-expectation-and-use-it-in-airflow-dag/719)

<div class="topic-metadata">

**Author:** [@shaharp](https://discourse.greatexpectations.io/u/shaharp)\
**Replies:** 2\
**Last updated:** [April 20, 2021, 2:57pm UTC](https://discourse.greatexpectations.io/t/create-customize-expectation-and-use-it-in-airflow-dag/719 "2021-04-20T14:57:31Z")

</div>

Hi everyone, It is possible to create a simple suite with only one expectation (let’s say it expect\_column\_values\_to\_not\_be\_null ) and then call the GreatExpectation Operator and send the column name as variable/paramet…

---

## [Format Run name (or batch id?) in Document Store (to be used in Airflow)](https://discourse.greatexpectations.io/t/format-run-name-or-batch-id-in-document-store-to-be-used-in-airflow/605)

<div class="topic-metadata">

**Author:** [@cityfish](https://discourse.greatexpectations.io/u/cityfish)\
**Replies:** 1\
**Last updated:** [February 19, 2021, 9:51pm UTC](https://discourse.greatexpectations.io/t/format-run-name-or-batch-id-in-document-store-to-be-used-in-airflow/605 "2021-02-19T21:51:44Z")

</div>

Context: We’re using shared Document “Store” on S3 for all our GE runs. We do run GE in Airflow, and I noticed that finding specific runs in the store is almost impossible. Here is the issue: We run Airflow jobs on a …

---

## [Write/Update Only S3-hosted Data Docs on S3](https://discourse.greatexpectations.io/t/write-update-only-s3-hosted-data-docs-on-s3/531)

<div class="topic-metadata">

**Author:** [@cityfish](https://discourse.greatexpectations.io/u/cityfish)\
**Replies:** 4\
**Last updated:** [December 4, 2020, 3:44am UTC](https://discourse.greatexpectations.io/t/write-update-only-s3-hosted-data-docs-on-s3/531 "2020-12-04T03:44:01Z")

</div>

Can someone explain better how does skip\_and\_clean\_missing works for DefaultSiteIndexBuilder . Do I understand correctly that if true, it will remove any html that has no related suite/expectation in the current config? …

---

## [Airflow Integration](https://discourse.greatexpectations.io/t/airflow-integration/34)

<div class="topic-metadata">

**Author:** [@jpcampbell42](https://discourse.greatexpectations.io/u/jpcampbell42)\
**Replies:** 3\
**Last updated:** [August 26, 2020, 8:45pm UTC](https://discourse.greatexpectations.io/t/airflow-integration/34 "2020-08-26T20:45:35Z")

</div>

I hear lots of questions from users who are integrating Great Expectations into existing pipelines that use Airflow to manage scheduling and execution. As part of a recent meetup, I built a minimal working airflow exampl…