How to configure a Pandas/filesystem Datasource

Archive
,
1

This article is for comments to: https://docs.greatexpectations.io/en/latest/how_to_guides/configuring_datasources/how_to_configure_a_pandas_filesystem_datasource.html

Please comment +1 if this How to is important to you.

2 Likes
2

Is there an example showing how to achieve something similar to SubdirReaderBatchKwargsGenerator? Should I use ConfiguredAssetFilesystemDataConnector or Inferred one? I have different files in one directory, there is not much pattern, each file itself is a data asset and their have their own format (csv/pickle). Is regex now the only way to configure data assets?

2 Likes
3

Can someone please help with this use case?

1 Like
4

Additionally, where can I adjust/insert the kwargs for file types and formats.
e.g.: pipe delimiters.

5

Anyone have latest/working link to above page? I’m trying to figure how to specify the delimiter to a file on the local FS. Thanks!

6

To get this working, I added below segment to batch_request in the expectations\mysuite.json

“batch_spec_passthrough”: {
“reader_method”: “read_csv”,
“reader_options”: {
“sep”: “|”
}
}