We are getting below error when we tried to configure Data Source and read data from s3 in pandas engine GE.
great_expectations.exceptions.exceptions.BatchKwargsError: Unable to build batch_kwargs. The asset may not be configured correctly. If s3 returned common prefixes it may not have been able to identify desired keys, and they are included in the incomplete batch_kwargs object returned with this error.
We tried below configuration but it gives above error
pandas_s3:
class_name: PandasDatasource
batch_kwargs_generators:
pandas_s3_generator:
class_name: S3GlobReaderBatchKwargsGenerator
bucket: xxx-xxx-xxx-xxxxxx-bucket
reader_method: read_csv
reader_options:
sep: “,”
delimiter: “/”
assets:
client_csv_file_test:
prefix: /xxx/xx-xx-xxxxx/xxxxx/
regex_filter: ‘/xxx/xx-xx-xxxxx/xxxxx/Client.*.csv’
module_name: great_expectations.datasource
data_asset_type:
class_name: PandasDataset
module_name: great_expectations.dataset
Can you please help here.