Hi, I am using GX version 1.2.4
My scenario is to validate a spark dataframe where some columns are array, but GX does not support validating array. My current approach is splitting those array columns into new dataframes, then creating an expectation suite and a validation definition for each one.
I gathered all validation definitions into a single checkpoint which looked like the following
Hi May! You can create distinct validation definitions for different DataFrames by adding them to your context and passing the corresponding validation definitions to the checkpoint. Example:
Hi Adeola. Thank you for your response! I understand how to configure checkpoint with multiple validation definitions, but I don’t know how to run the checkpoint. I’m currently passing a dataframe in a batch_parameters like the following:
Apologies for the confusion earlier-- you’re correct, what I described is only possible for non-DataFrame data sources. For DataFrames, a checkpoint requires a single set of batch parameters, so achieving this isn’t feasible with a single checkpoint.