BuildBatchRequestError when running a checkpoint (reading data from "dataframe")

GX Core Support
1

My GX version = 1.23.2.

In my script, I am reading data from Pandas Dataframe (df_bug, df_us & df_tc). Created the data_source using

data_source = context.data_sources.add_pandas(name=data_source_name)

The assets also got created correctly.

**** output **** 
Asset BUGS created.
Asset USERSTORYS created.
Asset TESTCASES created.

Assets dict: {'BUGS': DataFrameAsset(name='BUGS_asset', type='dataframe', id=UUID('eb06e0c2-20cd-44fe-a51a-178d91b36304'), order_by=[], batch_metadata={}, batch_definitions=[]), 'USERSTORYS': DataFrameAsset(name='USERSTORYS_asset', type='dataframe', id=UUID('9f10d93c-1978-42f1-b632-1f0a73a05ed2'), order_by=[], batch_metadata={}, batch_definitions=[]), 'TESTCASES': DataFrameAsset(name='TESTCASES_asset', type='dataframe', id=UUID('41e51c93-69c5-4867-beeb-374af59520fa'), order_by=[], batch_metadata={}, batch_definitions=[])}

And so did the batch - definition

**** output **** 
Batch Definition dict: {'BUGS': BatchDefinition(id=UUID('cf9cbd44-8b95-4b21-8a95-3699ccbcc052'), name='BUGS_batch_def', partitioner=None), 'USERSTORYS': BatchDefinition(id=UUID('eddf290f-fff5-4640-810a-761069eb4112'), name='USERSTORYS_batch_def', partitioner=None), 'TESTCASES': BatchDefinition(id=UUID('482fbd03-585c-41f5-8e70-9dd2eeaada2e'), name='TESTCASES_batch_def', partitioner=None)}

BUGS: (1003, 10)
USERSTORYS: (1000, 6)
TESTCASES: (1000, 6)

And so does the “Validation Definition”.

*** output ***
bugs_validation_definition
us_validation_definition
tc_validation_definition

However, when i run the “checkpoint” with these 3 validation definitions, I get the error

My code snippet for checkpoint:

# ============================
# Checkpoint
# ============================

# Define unique checkpoint name
checkpoint_name = "dq_checkpoint"

# Define action list
action_list = [
    # This Action updates the Data Docs static website with the Validation
    #   Results after the Checkpoint is run.
    UpdateDataDocsAction(
        name="update_all_data_docs",
    )
]

# Define checkpoint & add to context
try:
    checkpoint = context.checkpoints.add(
        gx.Checkpoint(
            name=checkpoint_name,
            validation_definitions=[
                                    bugs_val_def,    # BUGS validation definition
                                    us_val_def,     # USERSTORYS validation definition 
                                    tc_val_def     # TESTCASES validation definition
                                   ],
            actions=action_list,
            result_format={"result_format": "SUMMARY"},
        )
    )
except (gx.exceptions.DataContextError, ValueError):
    checkpoint = context.checkpoints.get(
            name = checkpoint_name
    )

# Verification
print(f"{checkpoint.name} created")

validation_results_chckpnt = checkpoint.run(
    batch_parameters={
        "BUGS_asset": {"dataframe": df_bug},
        "USERSTORYS_asset": {"dataframe": df_us},
        "TESTCASES_asset": {"dataframe": df_tc},
    },
    run_id=RunIdentifier(run_name="DQ_POC_Run")
)

Error I get

BuildBatchRequestError                    Traceback (most recent call last)
/tmp/ipykernel_58/1026562109.py in <cell line: 0>()
      4 #                                             )
      5 
----> 6 validation_results_chckpnt = checkpoint.run(
      7     batch_parameters={
      8   

.
.
.
.
.
BuildBatchRequestError: Bad input to build_batch_request: options must contain exactly 1 key, 'dataframe'.

So, how do I:

  • Create Checkpoint & run it as well when reading from pandas dataframe?

Note: Running the validation definition individually works fine

image

2

Hi @KK1259, thanks for the question. Checkpoint.run accepts a single set of batch parameters and passes that same dict to every validation definition in the checkpoint. There’s no per-asset routing, so your asset-keyed dict reaches each dataframe asset as three unknown keys and raises “options must contain exactly 1 key, ‘dataframe’”.

For multiple dataframes, that means one checkpoint per dataframe, each run with batch_parameters={"dataframe": df}. If you don’t need a checkpoint’s actions (data docs updates, notifications, etc.), ValidationDefinition.run(batch_parameters={"dataframe": df}) is simpler and runs the same validation.

This part of the API is under consideration for change, and a future release may let a checkpoint accept different batch parameters for each validation definition.

Others have hit the same limitation: #1876, #2015.

3

Thank you @joshstauffer , I guess I am a little late to discover this :grinning_face: