# How to load data from S3 for validation with Pandas as a batch

**URL:** <https://discourse.greatexpectations.io/t/how-to-load-data-from-s3-for-validation-with-pandas-as-a-batch/194>\
**Category:** Archive\
**Tags:** how-to, help-wanted\
**Created:** [May 27, 2020, 9:22pm UTC](https://discourse.greatexpectations.io/t/how-to-load-data-from-s3-for-validation-with-pandas-as-a-batch/194 "2020-05-27T21:22:21Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kyle](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/kyle/32/18_2.png) [@kyle](https://discourse.greatexpectations.io/u/kyle)\
**Post date:** [May 27, 2020, 9:22pm UTC](https://discourse.greatexpectations.io/t/how-to-load-data-from-s3-for-validation-with-pandas-as-a-batch/194/1 "2020-05-27T21:22:21Z")

</div>

This article is for comments to: [https://docs.greatexpectations.io/en/latest/how\_to\_guides/creating\_batches/how\_to\_load\_data\_from\_s3\_for\_validation\_with\_pandas\_as\_a\_batch.html](https://docs.greatexpectations.io/en/latest/how_to_guides/creating_batches/how_to_load_data_from_s3_for_validation_with_pandas_as_a_batch.html)

Please comment +1 if this How to is important to you.

---

<div class="post-metadata">

**Author:** ![krisp](https://avatars.discourse-cdn.com/v4/letter/k/e95f7d/32.png) [@krisp](https://discourse.greatexpectations.io/u/krisp)\
**Post date:** [May 28, 2020, 2:54am UTC](https://discourse.greatexpectations.io/t/how-to-load-data-from-s3-for-validation-with-pandas-as-a-batch/194/2 "2020-05-28T02:54:37Z")

</div>

+1

I don’t see how to define s3 datasource in 0.11.0 so that it works. I have (multiple versions of):

```auto
  devices_s3:
    batch_kwargs_generators:
      s3gen:
        class_name: S3GlobReaderBatchKwargsGenerator
        bucket: mybucket/
        reader_method: parquet
        assets: 
          device_ip_sample: 
            prefix: device_ip_sample/
            regex_filter: .*
            dictionary_assets: True
    module_name: great_expectations.datasource
    data_asset_type:
      module_name: great_expectations.dataset
      class_name: SparkDFDataset
    class_name: SparkDFDatasource

```

but `great_expectations datasource profile devices_s3` gives me this error:

```auto
Unrecognized batch_parameter(s): {'data_asset_name'}
(...omitted errors...)
  File "/home/kris/anaconda3/lib/python3.7/site-packages/great_expectations/datasource/batch_kwargs_generator/s3_batch_kwargs_generator.py", line 208, in _build_batch_kwargs
    "s3": "s3a://" + self.bucket + "/" + key,
TypeError: can only concatenate str (not "dict") to str

```

Thanks for your help!

---

<div class="post-metadata">

**Author:** ![aylr](https://avatars.discourse-cdn.com/v4/letter/a/8797f3/32.png) [@aylr](https://discourse.greatexpectations.io/u/aylr)\
**Post date:** [May 28, 2020, 10:37pm UTC](https://discourse.greatexpectations.io/t/how-to-load-data-from-s3-for-validation-with-pandas-as-a-batch/194/3 "2020-05-28T22:37:25Z")

</div>

@krisp we’ll work on reproducing this! Thanks for the report. Just to confirm is this on version 0.11.0?

---

<div class="post-metadata">

**Author:** ![jpcampbell42](https://avatars.discourse-cdn.com/v4/letter/j/838e76/32.png) [@jpcampbell42](https://discourse.greatexpectations.io/u/jpcampbell42)\
**Post date:** [June 24, 2020, 9:57pm UTC](https://discourse.greatexpectations.io/t/how-to-load-data-from-s3-for-validation-with-pandas-as-a-batch/194/4 "2020-06-24T21:57:06Z")

</div>

@krisp -\> Have you confirmed the version by chance? I believe this was an issue that is now fixed…