# CSVAsset only showing one Batch even though batching\_regex matches two files

**URL:** https://discourse.greatexpectations.io/t/csvasset-only-showing-one-batch-even-though-batching-regex-matches-two-files/1296
**Category:** GX Core Support
**Created:** [July 20, 2023, 10:06pm UTC](https://discourse.greatexpectations.io/t/csvasset-only-showing-one-batch-even-though-batching-regex-matches-two-files/1296 "2023-07-20T22:06:43Z")
**Posts on this page:** 12
**Page:** 1

<div class="post-metadata">

### Author: ![Barrett](https://avatars.discourse-cdn.com/v4/letter/b/54ee81/32.png) [@Barrett](https://discourse.greatexpectations.io/u/Barrett)
#### Post date: [July 20, 2023, 10:06pm UTC](https://discourse.greatexpectations.io/t/csvasset-only-showing-one-batch-even-though-batching-regex-matches-two-files/1296/1 "2023-07-20T22:06:43Z")

</div>

I need some pointers as to what I’m doing wrong, please.

Following the [process here](https://docs.greatexpectations.io/docs/guides/connecting_to_your_data/fluent/batch_requests/how_to_request_data_from_a_data_asset/) and in [docs for setting up a multi-file Pandas filesystem datasource](https://docs.greatexpectations.io/docs/guides/connecting_to_your_data/fluent/filesystem/connect_filesystem_source_data), I’ve created a datasource and a CSVAsset with a batching\_regex which I’ve verified matches two files in the same data directory.

When then running a checkpoint, I was only seeing one file processed. In working backward to see what’s wrong, I’m finding that only one file is listed when i run

```auto
mybatchrequest = asset.build_batch_request()
for batch in asset.get_batch_list_from_batch_request(mybatchrequest):
  print(batch.batch_spec)

```

There’s not something special that has to be done to get all the files when the regex includes group names, is there? My regex in the asset is `batching_regex=re.compile('customer_(?P<datetime>\\d{14})\\.csv')`

Are there other likely sources of this that I should be looking into?

---

<div class="post-metadata">

### Author: ![CesarGarcia](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/cesargarcia/32/195_2.png) [@CesarGarcia](https://discourse.greatexpectations.io/u/CesarGarcia)
#### Post date: [July 20, 2023, 10:48pm UTC](https://discourse.greatexpectations.io/t/csvasset-only-showing-one-batch-even-though-batching-regex-matches-two-files/1296/2 "2023-07-20T22:48:14Z")

</div>

Hi Barret,

The format looks ok to me. The only thing I find strange is the re.compile when defining the batch regex.

According to the [Batch Request documentation](https://docs.greatexpectations.io/docs/terms/batch_request/), you define the regex as:

```python
asset = datasource.add_csv_asset(
    name="csv_asset",
    batching_regex=r"yellow_tripdata_sample_(?P<year>\d{4})-(?P<month>\d{2}).csv",
    order_by=["year", "month"],
)

```

So instead of passing it as re.compile, could you try passing it as an r string?

Another thing to consider is the way you are building your batch request. According to the docs it’s done using a call like this:

```python
batch_request = asset.build_batch_request(options={"year": "2019", "month": "02"})

```

In you code, you define the group based on datetime, but then in the asset.build\_batch\_request, you are not passing the datetime as an option.

So as a general rule, first you define the regex using the r’string, defining the group “tag”. Then you can select which of the matching files to use, either all or one specific file.

As a sidetone, as I was playing around batch requests, I discovered that when defining a batch request you can get a single file, or the complete list of elements if you don’t pass any argument. For example, given a dir with 12 monthly files, you can’t just pass any regex or list as arguments to the call of build\_batch\_request() (So you can’t pass options={ “month”: ["01,“02”, "03]} or “month”: “^1\d\*” (to choose all months starting by 1))

I hope it helps!  
César

---

<div class="post-metadata">

### Author: ![Barrett](https://avatars.discourse-cdn.com/v4/letter/b/54ee81/32.png) [@Barrett](https://discourse.greatexpectations.io/u/Barrett)
#### Post date: [July 21, 2023, 7:15am UTC](https://discourse.greatexpectations.io/t/csvasset-only-showing-one-batch-even-though-batching-regex-matches-two-files/1296/3 "2023-07-21T07:15:35Z")

</div>

I think I’ve figured this out. After rebuilding all the code from scratch, what I’m seeing is that the content of a batch request is not changed when a batch request is recreated after the content of the file system changes.

To replicate:

1. Create a fielsystem datasource and asset using a batching\_regex which will match some number of existing files.
2. Generate a batch request and print the batches in the request. Observe the count matches the number of files.
3. Add an additional file matching the batching\_regex parameter to the filesystem
4. Generate a new batch request and prtint the batches in the request. Observe the new file is not listed in the batch.

Is that expected behavior? I thought I understood that the batching\_regex was re-evaulated when each time you created a batch request.

[edit]  
The opposite case is apparently also true: If you create an asset which includes some number of files and create a batch request from that asset, then remove one or more of the files and create a new batch request, running the code below raises a `FileNotFoundError: [Errno 2] No such file or directory: <the removed file>`

```auto
for batch in asset.get_batch_list_from_batch_request(mybatchrequest):
  print(batch.batch_spec)

```

---

<div class="post-metadata">

### Author: ![HaebichanGX](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/haebichangx/32/211_2.png) [@HaebichanGX](https://discourse.greatexpectations.io/u/HaebichanGX)
#### Post date: [July 21, 2023, 1:44pm UTC](https://discourse.greatexpectations.io/t/csvasset-only-showing-one-batch-even-though-batching-regex-matches-two-files/1296/4 "2023-07-21T13:44:58Z")

</div>

Hi @Barrett so this is a common issue raised raised by our users. We’ve just worked on a GX-recommend method for processing multiple batches using one checkpoint. The docs are here:

> **[Validate multiple Batches from a Batch Request with a single Checkpoint |...](https://docs.greatexpectations.io/docs/guides/validation/checkpoints/how_to_validate_multiple_batches_within_single_checkpoint/)**
>
> By default, a Checkpoint only validates the last Batch included in a Batch Request. Use the information provided here to learn how you can use a Python loop and the Checkpoint validations parameter to validate multiple Batches identified by a single...

---

<div class="post-metadata">

### Author: ![Barrett](https://avatars.discourse-cdn.com/v4/letter/b/54ee81/32.png) [@Barrett](https://discourse.greatexpectations.io/u/Barrett)
#### Post date: [July 21, 2023, 3:17pm UTC](https://discourse.greatexpectations.io/t/csvasset-only-showing-one-batch-even-though-batching-regex-matches-two-files/1296/5 "2023-07-21T15:17:10Z")

</div>

Thank you, @HaebichanGX !

---

<div class="post-metadata">

### Author: ![Barrett](https://avatars.discourse-cdn.com/v4/letter/b/54ee81/32.png) [@Barrett](https://discourse.greatexpectations.io/u/Barrett)
#### Post date: [July 21, 2023, 5:38pm UTC](https://discourse.greatexpectations.io/t/csvasset-only-showing-one-batch-even-though-batching-regex-matches-two-files/1296/6 "2023-07-21T17:38:41Z")

</div>

Actually, @HaebichanGX , follow up question, if I may:

The solution you linked to covers getting the validation to run against all the files in the batch, but it doesn’t address the issue that new files added to the file system are not recognized by the batch even after generating a new request. The only way I’ve been able to solve for that is to create a new Asset.

Shouldn’t a batch request find all the files matching the batching\_regex of the Asset at the time the batch request is created?

---

<div class="post-metadata">

### Author: ![vipinadm](https://avatars.discourse-cdn.com/v4/letter/v/ecd19e/32.png) [@vipinadm](https://discourse.greatexpectations.io/u/vipinadm)
#### Post date: [July 22, 2023, 5:36am UTC](https://discourse.greatexpectations.io/t/csvasset-only-showing-one-batch-even-though-batching-regex-matches-two-files/1296/7 "2023-07-22T05:36:34Z")

</div>

Hi @HaebichanGX , Validating multiple batches within Single checking point using batch request and validation list is very good improvement. However it still come with a challenge and I think it will be good point for another feature in GX.  
Validation with multiple batches will create separate validation results for each batch. Mostly we will have relevant data from data source in the same checkpoint. It will be good to have option of consolidated single validation results. Why would one will like to see many entries in data docs site for same expectations?

Thoughts ? will GX architecture will allow this?

---

<div class="post-metadata">

### Author: ![HaebichanGX](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/haebichangx/32/211_2.png) [@HaebichanGX](https://discourse.greatexpectations.io/u/HaebichanGX)
#### Post date: [July 24, 2023, 4:12pm UTC](https://discourse.greatexpectations.io/t/csvasset-only-showing-one-batch-even-though-batching-regex-matches-two-files/1296/8 "2023-07-24T16:12:43Z")

</div>

Hi @vipinadm, first, welcome to GX discourse! Yes I understand your point. So for that specific use case, my suggestion is to treat it during the asset level. So let’s say you’re working with a SQL datasource and have multiple data. You can use the **add\_query\_asset** to merge the data (union or otherwise) to create one mega batch. Then you can run one expectation against one mega batch of data. For filesystem datasource, the process would be something similar.

The example we gave to match one expectation suite to multiple batch requests inside a validations list was to keep things simple in terms of code and explanation and to focus on checkpoints. I hope that clarifies our intentions and the solution to the potential issue you raised.

---

<div class="post-metadata">

### Author: ![slack\_user](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/slack_user/32/202_2.png) [@slack\_user](https://discourse.greatexpectations.io/u/slack_user)
#### Post date: [July 24, 2023, 4:35pm UTC](https://discourse.greatexpectations.io/t/csvasset-only-showing-one-batch-even-though-batching-regex-matches-two-files/1296/9 "2023-07-24T16:35:10Z")

</div>

Hi @HaebichanGX, Could you please shared an example of add query asset. I am reading data from GCS bucket  
Note: `@Vipin Admulwar` originally [posted this reply in Slack](https://greatexpectationstalk.slack.com/archives/CUTCNHN82/p1690216501401229?thread_ts=1689890829.042559&cid=CUTCNHN82). It might not have transferred perfectly.

---

<div class="post-metadata">

### Author: ![slack\_user](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/slack_user/32/202_2.png) [@slack\_user](https://discourse.greatexpectations.io/u/slack_user)
#### Post date: [July 24, 2023, 4:39pm UTC](https://discourse.greatexpectations.io/t/csvasset-only-showing-one-batch-even-though-batching-regex-matches-two-files/1296/10 "2023-07-24T16:39:08Z")

</div>

… or a hql ( Hive metastore ) example will be helpful  
Note: `@Vipin Admulwar` originally [posted this reply in Slack](https://greatexpectationstalk.slack.com/archives/CUTCNHN82/p1690216740971889?thread_ts=1689890829.042559&cid=CUTCNHN82). It might not have transferred perfectly.

---

<div class="post-metadata">

### Author: ![HaebichanGX](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/haebichangx/32/211_2.png) [@HaebichanGX](https://discourse.greatexpectations.io/u/HaebichanGX)
#### Post date: [July 24, 2023, 4:53pm UTC](https://discourse.greatexpectations.io/t/csvasset-only-showing-one-batch-even-though-batching-regex-matches-two-files/1296/11 "2023-07-24T16:53:30Z")

</div>

Hi @vipinadm there is no doc on hive metastore. The doc on add query asset is here: [Manage SQL Data Assets | Great Expectations](https://docs.greatexpectations.io/docs/guides/connecting_to_your_data/fluent/database/sql_data_assets)

You can use this advanced GX flow chart to visualize the workflow: [GX Advanced Used Case Flow Chart (INCOMPLETE) | Lucidspark](https://lucid.app/lucidspark/c8d32b16-7d66-4fcf-9215-bf6b5e1d5c4e/edit?invitationId=inv_273a703d-edc8-4bf5-95cb-bec9f81b11db&referringApp=slack&page=0_0#)

---

<div class="post-metadata">

### Author: ![vipinadm](https://avatars.discourse-cdn.com/v4/letter/v/ecd19e/32.png) [@vipinadm](https://discourse.greatexpectations.io/u/vipinadm)
#### Post date: [July 25, 2023, 4:54am UTC](https://discourse.greatexpectations.io/t/csvasset-only-showing-one-batch-even-though-batching-regex-matches-two-files/1296/12 "2023-07-25T04:54:18Z")

</div>

Hi @HaebichanGX, Can you please confirm hive datasource supported in GX? I am using spark to read the data from GCS bucket so add\_query\_asset won’t help.