# Can GE access / validate data from Spark, stored in an S3 bucket?

**URL:** https://discourse.greatexpectations.io/t/can-ge-access-validate-data-from-spark-stored-in-an-s3-bucket/704
**Category:** Archive
**Tags:** s3
**Created:** [April 1, 2021, 1:28pm UTC](https://discourse.greatexpectations.io/t/can-ge-access-validate-data-from-spark-stored-in-an-s3-bucket/704 "2021-04-01T13:28:20Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![alexc](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@alexc](https://discourse.greatexpectations.io/u/alexc)
#### Post date: [April 1, 2021, 1:28pm UTC](https://discourse.greatexpectations.io/t/can-ge-access-validate-data-from-spark-stored-in-an-s3-bucket/704/1 "2021-04-01T13:28:20Z")

</div>

I’d like to be able to run Great Expectations at several steps in an airflow pipeline. Between which, Spark is being used used to cleanse / transform data and store parquet files in S3 buckets. Can Great Expectations access these as a data source? I can find only references to Spark on filesystem, and S3 in conjunction with Pandas.

Any help appreciated.

Thanks,  
Alex.

---

<div class="post-metadata">

### Author: ![eugene.mandel](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/eugene.mandel/32/22_2.png) [@eugene.mandel](https://discourse.greatexpectations.io/u/eugene.mandel)
#### Post date: [April 20, 2021, 3:14pm UTC](https://discourse.greatexpectations.io/t/can-ge-access-validate-data-from-spark-stored-in-an-s3-bucket/704/2 "2021-04-20T15:14:21Z")

</div>

@alexc Yes, you can validate a Parquet file in an S3 bucket as a step in your Airflow DAG. We will create a documentation article for this case, but in the meantime,

1. please use this article to see how to validate a Spark dataframe: [How to load a Spark DataFrame as a Batch — great\_expectations documentation](https://docs.greatexpectations.io/en/latest/guides/how_to_guides/creating_batches/how_to_load_a_spark_dataframe_as_a_batch.html)
2. Replace the df in runtime\_parameters with {“path”: “s3://my.s3.path”}