# Profile data on S3 in multiple prefixes

**URL:** <https://discourse.greatexpectations.io/t/profile-data-on-s3-in-multiple-prefixes/397>\
**Category:** Archive\
**Created:** [September 21, 2020, 1:58pm UTC](https://discourse.greatexpectations.io/t/profile-data-on-s3-in-multiple-prefixes/397 "2020-09-21T13:58:38Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![falizadeh](https://avatars.discourse-cdn.com/v4/letter/f/f14d63/32.png) [@falizadeh](https://discourse.greatexpectations.io/u/falizadeh)\
**Post date:** [September 21, 2020, 1:58pm UTC](https://discourse.greatexpectations.io/t/profile-data-on-s3-in-multiple-prefixes/397/1 "2020-09-21T13:58:38Z")

</div>

We are using GE to profile our ingested datasets on S3. It mostly works but we have an issue with the scenario where data is ingested in multiple prefixes. I am wondering how I can specify the prefixes in my regex\_filter. This is what I have:  
“assets”: {  
“name\_1”: {  
“prefix”: “prefix\_1/”,  
“regex\_filter”: “prefix\_1/exportdate=.\*/.\*parquet”  
}  
}  
And I would like to profile these files from these prefixes:  
prefix\_1/exportdate=2020/1.parquet  
prefix\_1/exportdate=2020/2.parquet  
prefix\_1/exportdate=2019/10.parquet  
prefix\_1/exportdate=2019/20.parquet

But I get this error: great\_expectations.exceptions.BatchKwargsError: Unrecognized batch\_kwargs for spark\_source

---

<div class="post-metadata">

**Author:** ![bhcastleton](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/bhcastleton/32/73_2.png) [@bhcastleton](https://discourse.greatexpectations.io/u/bhcastleton)\
**Post date:** [September 21, 2020, 2:13pm UTC](https://discourse.greatexpectations.io/t/profile-data-on-s3-in-multiple-prefixes/397/2 "2020-09-21T14:13:03Z")

</div>

@falizadeh - Clarifying question:  
Do these files have the same schema so you want them all to be profiled together and eventually tested in a single batch?

---

<div class="post-metadata">

**Author:** ![falizadeh](https://avatars.discourse-cdn.com/v4/letter/f/f14d63/32.png) [@falizadeh](https://discourse.greatexpectations.io/u/falizadeh)\
**Post date:** [September 21, 2020, 2:34pm UTC](https://discourse.greatexpectations.io/t/profile-data-on-s3-in-multiple-prefixes/397/3 "2020-09-21T14:34:49Z")

</div>

@bhcastleton yes, they all have the same schemas only they are placed in different prefixes.

---

<div class="post-metadata">

**Author:** ![bhcastleton](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/bhcastleton/32/73_2.png) [@bhcastleton](https://discourse.greatexpectations.io/u/bhcastleton)\
**Post date:** [September 21, 2020, 2:41pm UTC](https://discourse.greatexpectations.io/t/profile-data-on-s3-in-multiple-prefixes/397/4 "2020-09-21T14:41:52Z")

</div>

@falizadeh - Thanks. Unfortunately, the current answer is that Great Expectations doesn’t have a native method to handle this internally unless the computing framework you’re working in can handle it. For example, this post talks about how to use Spark to load multiple files into one batch. [Validate foreign keys / load multiple files to a single spark dataframe with batch generator](https://discourse.greatexpectations.io/t/validate-foreign-keys-load-multiple-files-to-a-single-spark-dataframe-with-batch-generator/278)

Spark supports specifying a path with multiple files and spark will read them into a data frame. So this works. Other frameworks may not support this. Great Expectations currently doesn’t handle that internally so you’d have to get all the files into a single file or dataframe before pointing Great Expectations to it.