# \#s3

**URL:** https://discourse.greatexpectations.io/tag/s3/5.md

[Latest](https://discourse.greatexpectations.io/latest.md) · [Categories](https://discourse.greatexpectations.io/categories.md) · [Tags](https://discourse.greatexpectations.io/tags.md)

---

## [GE run fails on AWS Airflow with File System \[Errno 30\] Read-only file system: '/usr/local/airflow/dags/data-quality/gx'; 3080](https://discourse.greatexpectations.io/t/ge-run-fails-on-aws-airflow-with-file-system-errno-30-read-only-file-system-usr-local-airflow-dags-data-quality-gx-3080/1704)

<div class="topic-metadata">

**Author:** [@zfromsyd](https://discourse.greatexpectations.io/u/zfromsyd)\
**Replies:** 6\
**Last updated:** [July 16, 2025, 5:20pm UTC](https://discourse.greatexpectations.io/t/ge-run-fails-on-aws-airflow-with-file-system-errno-30-read-only-file-system-usr-local-airflow-dags-data-quality-gx-3080/1704 "2025-07-16T17:20:58Z")

</div>

Hi, We are upgrading Great Expectations to 0.18.10. We are currently experiencing a blocker issue from one of the packages that throws a read-only file system error. great-expectations version 0.18.10 airflow-provider…

---

## [GX Core only reads last file in directory when using add\_batch\_definition\_whole\_directory()](https://discourse.greatexpectations.io/t/gx-core-only-reads-last-file-in-directory-when-using-add-batch-definition-whole-directory/2201)

<div class="topic-metadata">

**Author:** [@lcoogan](https://discourse.greatexpectations.io/u/lcoogan)\
**Replies:** 3\
**Last updated:** [June 18, 2025, 5:10pm UTC](https://discourse.greatexpectations.io/t/gx-core-only-reads-last-file-in-directory-when-using-add-batch-definition-whole-directory/2201 "2025-06-18T17:10:43Z")

</div>

Hello, I have an s3 prefix containing a set of .parquet files. I’m trying to use GX Core to read these files into a single batch to do some validations. I’m using add\_batch\_definition\_whole\_directory(), because the doc…

---

## [Issue with checkpoint.run() results when we have multiple validation definitions](https://discourse.greatexpectations.io/t/issue-with-checkpoint-run-results-when-we-have-multiple-validation-definitions/2119)

<div class="topic-metadata">

**Author:** [@vijaykumar](https://discourse.greatexpectations.io/u/vijaykumar)\
**Replies:** 1\
**Last updated:** [April 1, 2025, 2:09pm UTC](https://discourse.greatexpectations.io/t/issue-with-checkpoint-run-results-when-we-have-multiple-validation-definitions/2119 "2025-04-01T14:09:59Z")

</div>

Hi Everyone :slightly\_smiling\_face:, I am using checkpoint which have multiple validation definitions , each validation definitions represent one batch. when i run results= checkpoint.run() it is internally processing …

---

## [Unable to configure the GE checks to validate parquet files on s3](https://discourse.greatexpectations.io/t/unable-to-configure-the-ge-checks-to-validate-parquet-files-on-s3/2141)

<div class="topic-metadata">

**Author:** [@sourabh.somvanshi0](https://discourse.greatexpectations.io/u/sourabh.somvanshi0)\
**Replies:** 1\
**Last updated:** [March 31, 2025, 3:44pm UTC](https://discourse.greatexpectations.io/t/unable-to-configure-the-ge-checks-to-validate-parquet-files-on-s3/2141 "2025-03-31T15:44:15Z")

</div>

I am currently accessing parquet files from aws s3 and using AWS MWAA (Airflow) to validate input files. I am using the dataframe approach however I want to access the files directly from s3 without reading into data fra…

---

## [Fluent datasources bucket env variable substitution](https://discourse.greatexpectations.io/t/fluent-datasources-bucket-env-variable-substitution/1689)

<div class="topic-metadata">

**Author:** [@Stix](https://discourse.greatexpectations.io/u/Stix)\
**Replies:** 1\
**Last updated:** [April 16, 2024, 5:25pm UTC](https://discourse.greatexpectations.io/t/fluent-datasources-bucket-env-variable-substitution/1689 "2024-04-16T17:25:09Z")

</div>

Hello guys, I am using a fluent datasource of type pandas\_s3. The problem is that when I run checkpoint the bucket variable it is not replaced and it gives me this error: File “/usr/local/lib/python3.9/site-packages/b…

---

## [Help understanding datasource\_name config for RuntimeBatchRequest](https://discourse.greatexpectations.io/t/help-understanding-datasource-name-config-for-runtimebatchrequest/1374)

<div class="topic-metadata">

**Author:** [@anais](https://discourse.greatexpectations.io/u/anais)\
**Replies:** 5\
**Last updated:** [September 14, 2023, 5:58pm UTC](https://discourse.greatexpectations.io/t/help-understanding-datasource-name-config-for-runtimebatchrequest/1374 "2023-09-14T17:58:34Z")

</div>

Hi, I am trying to use great expectations with aws glue. The data will be read at runtime, I follow this tutorial How to Use Great Expectations in AWS Glue | Great Expectations My problem is to understand what to set fo…

---

## [Can a single expectation suite contain multiple tables with different expectations for different tables](https://discourse.greatexpectations.io/t/can-a-single-expectation-suite-contain-multiple-tables-with-different-expectations-for-different-tables/1301)

<div class="topic-metadata">

**Author:** [@lionel](https://discourse.greatexpectations.io/u/lionel)\
**Replies:** 1\
**Last updated:** [July 26, 2023, 2:59pm UTC](https://discourse.greatexpectations.io/t/can-a-single-expectation-suite-contain-multiple-tables-with-different-expectations-for-different-tables/1301 "2023-07-26T14:59:07Z")

</div>

I want to run GE against snowflake tables in a DB can i have one suite that specifies a expectation for a particular table. thanks

---

## [Can GX read data from S3?](https://discourse.greatexpectations.io/t/can-gx-read-data-from-s3/1261)

<div class="topic-metadata">

**Author:** [@austin\_gx](https://discourse.greatexpectations.io/u/austin_gx)\
**Replies:** 1\
**Last updated:** [July 14, 2023, 7:46pm UTC](https://discourse.greatexpectations.io/t/can-gx-read-data-from-s3/1261 "2023-07-14T19:46:02Z")

</div>

We have a ETL pipeline which stores the results in a S3 bucket and would like to understand if we can use the great\_expectations library to validate that the content present in the bucket is as per expectations. If this …

---

## [Hosting data docs on AWS S3](https://discourse.greatexpectations.io/t/hosting-data-docs-on-aws-s3/1059)

<div class="topic-metadata">

**Author:** [@chsafouane](https://discourse.greatexpectations.io/u/chsafouane)\
**Replies:** 1\
**Last updated:** [September 13, 2022, 6:58am UTC](https://discourse.greatexpectations.io/t/hosting-data-docs-on-aws-s3/1059 "2022-09-13T06:58:10Z")

</div>

Hey Everyone I would like to share Data Docs of my project on Amazon S3. The documentation says that I have to specify a bucket but I would like to place the Data Docs inside a specific folder of the bucket that I use. …

---

## [Increased execution time for expectations in Airflow](https://discourse.greatexpectations.io/t/increased-execution-time-for-expectations-in-airflow/1035)

<div class="topic-metadata">

**Author:** [@vhernandezs](https://discourse.greatexpectations.io/u/vhernandezs)\
**Replies:** 1\
**Last updated:** [May 10, 2022, 12:45pm UTC](https://discourse.greatexpectations.io/t/increased-execution-time-for-expectations-in-airflow/1035 "2022-05-10T12:45:08Z")

</div>

Dear Team. I am currently facing an issue while working with GreatExpectations in Airflow. In my current setup, I define several Airflow tasks to run GreatExpectations Checkpoints (version 0.13.37) to validate my data, …

---

## [Spark Connect to S3](https://discourse.greatexpectations.io/t/spark-connect-to-s3/925)

<div class="topic-metadata">

**Author:** [@xavy](https://discourse.greatexpectations.io/u/xavy)\
**Replies:** 0\
**Last updated:** [November 9, 2021, 1:09am UTC](https://discourse.greatexpectations.io/t/spark-connect-to-s3/925 "2021-11-09T01:09:21Z")

</div>

Hi All, I’m using InferredAssetS3DataConnector to connect to S3, but it is not working returning a message “ValueError: S3 query may not have been configured correctly.” After debug your code I found a bug related to …

---

## [Except\_file\_to\_exist not working](https://discourse.greatexpectations.io/t/except-file-to-exist-not-working/907)

<div class="topic-metadata">

**Author:** [@manojpandey](https://discourse.greatexpectations.io/u/manojpandey)\
**Replies:** 0\
**Last updated:** [October 20, 2021, 7:03pm UTC](https://discourse.greatexpectations.io/t/except-file-to-exist-not-working/907 "2021-10-20T19:03:40Z")

</div>

I am trying to use the except\_file\_to\_exist via FileDataAsset and it’s failing as this expectation is not loaded into registry. Failed at registry.py \_registered\_expectation.get(expectation\_name) I followed the blog G…

---

## [Creating Data source for s3 with pandas](https://discourse.greatexpectations.io/t/creating-data-source-for-s3-with-pandas/773)

<div class="topic-metadata">

**Author:** [@Garima93](https://discourse.greatexpectations.io/u/Garima93)\
**Replies:** 1\
**Last updated:** [June 14, 2021, 3:28pm UTC](https://discourse.greatexpectations.io/t/creating-data-source-for-s3-with-pandas/773 "2021-06-14T15:28:31Z")

</div>

We are getting below error when we tried to configure Data Source and read data from s3 in pandas engine GE. great\_expectations.exceptions.exceptions.BatchKwargsError: Unable to build batch\_kwargs. The asset may not be …

---

## [AttributeError: 'NoneType' object has no attribute 'sc'](https://discourse.greatexpectations.io/t/attributeerror-nonetype-object-has-no-attribute-sc/671)

<div class="topic-metadata">

**Author:** [@rakesh](https://discourse.greatexpectations.io/u/rakesh)\
**Replies:** 3\
**Last updated:** [May 19, 2021, 9:19pm UTC](https://discourse.greatexpectations.io/t/attributeerror-nonetype-object-has-no-attribute-sc/671 "2021-05-19T21:19:07Z")

</div>

I am trying to execute pyspark script via emr. Script will process files from S3 bucket and put into another folder. Code was working fine till march first week. But suddenly getting error at the initial phase where data…

---

## [How to install GE in AWS](https://discourse.greatexpectations.io/t/how-to-install-ge-in-aws/748)

<div class="topic-metadata">

**Author:** [@Garima93](https://discourse.greatexpectations.io/u/Garima93)\
**Replies:** 0\
**Last updated:** [May 11, 2021, 5:45am UTC](https://discourse.greatexpectations.io/t/how-to-install-ge-in-aws/748 "2021-05-11T05:45:27Z")

</div>

How can we install GE in AWS either in Lambda or in EC2. I can see below article which talks about as how to host and share data docs in S3. is it something that GE is still installed in local server only and the valid…

---

## [S3 Data Source Configures Successfully But Suite New Fails with New S3 Datasource](https://discourse.greatexpectations.io/t/s3-data-source-configures-successfully-but-suite-new-fails-with-new-s3-datasource/727)

<div class="topic-metadata">

**Author:** [@emekdahl-fetch](https://discourse.greatexpectations.io/u/emekdahl-fetch)\
**Replies:** 2\
**Last updated:** [May 3, 2021, 6:07pm UTC](https://discourse.greatexpectations.io/t/s3-data-source-configures-successfully-but-suite-new-fails-with-new-s3-datasource/727 "2021-05-03T18:07:11Z")

</div>

Following the instructions here on configuring an s3 data store gave me a success message but creating a new suite based on the s3 data source returns the following error: if len(value\["names"\]) \> 0: TypeError: list…

---

## [Can GE access / validate data from Spark, stored in an S3 bucket?](https://discourse.greatexpectations.io/t/can-ge-access-validate-data-from-spark-stored-in-an-s3-bucket/704)

<div class="topic-metadata">

**Author:** [@alexc](https://discourse.greatexpectations.io/u/alexc)\
**Replies:** 1\
**Last updated:** [April 20, 2021, 3:14pm UTC](https://discourse.greatexpectations.io/t/can-ge-access-validate-data-from-spark-stored-in-an-s3-bucket/704 "2021-04-20T15:14:21Z")

</div>

I’d like to be able to run Great Expectations at several steps in an airflow pipeline. Between which, Spark is being used used to cleanse / transform data and store parquet files in S3 buckets. Can Great Expectations acc…

---

## [Configuring S3 as Datastore](https://discourse.greatexpectations.io/t/configuring-s3-as-datastore/698)

<div class="topic-metadata">

**Author:** [@bradleyfay](https://discourse.greatexpectations.io/u/bradleyfay)\
**Replies:** 2\
**Last updated:** [April 1, 2021, 7:41pm UTC](https://discourse.greatexpectations.io/t/configuring-s3-as-datastore/698 "2021-04-01T19:41:26Z")

</div>

Hi All, I’m trying to configure an S3 bucket as a datastore. I’m following this guide: How to configure a Pandas/S3 Datasource — great\_expectations documentation, but running into errors. Specifically, “If s3 returned co…

---

## [Profiling overwriting validations and vice versa](https://discourse.greatexpectations.io/t/profiling-overwriting-validations-and-vice-versa/568)

<div class="topic-metadata">

**Author:** [@Gabbar](https://discourse.greatexpectations.io/u/Gabbar)\
**Replies:** 2\
**Last updated:** [December 18, 2020, 8:02am UTC](https://discourse.greatexpectations.io/t/profiling-overwriting-validations-and-vice-versa/568 "2020-12-18T08:02:55Z")

</div>

Hi background: I am running GE in hosted environment using this - DataContext created during the run. I have two different Jobs - one for Validation (daily) and another for Profiling(weekly/monthly). I am using same S…

---

## [Write/Update Only S3-hosted Data Docs on S3](https://discourse.greatexpectations.io/t/write-update-only-s3-hosted-data-docs-on-s3/531)

<div class="topic-metadata">

**Author:** [@cityfish](https://discourse.greatexpectations.io/u/cityfish)\
**Replies:** 4\
**Last updated:** [December 4, 2020, 3:44am UTC](https://discourse.greatexpectations.io/t/write-update-only-s3-hosted-data-docs-on-s3/531 "2020-12-04T03:44:01Z")

</div>

Can someone explain better how does skip\_and\_clean\_missing works for DefaultSiteIndexBuilder . Do I understand correctly that if true, it will remove any html that has no related suite/expectation in the current config? …

---

## [Databricks and AWS s3](https://discourse.greatexpectations.io/t/databricks-and-aws-s3/400)

<div class="topic-metadata">

**Author:** [@anthony](https://discourse.greatexpectations.io/u/anthony)\
**Replies:** 0\
**Last updated:** [September 21, 2020, 9:18pm UTC](https://discourse.greatexpectations.io/t/databricks-and-aws-s3/400 "2020-09-21T21:18:40Z")

</div>

We have noticed that there is some confusion around writing to s3 from a Databricks environment (e.g. writing to metadata stores / data docs). That is understandable as it’s a bit involved to set up access to s3 from wit…