Can I use a Spark Dataframe with Checkpoints?

Archive
1

Hi,

can I use a runtime batch request I defined with a spark data frame with checkpoints? Or is this currently unsupported?

The configuration I have works fine with validation operators but I could not figure out with the documentation how / if this could be done with the newer checkpoints.

I followed this guide for V3 to define the batch request:

https://docs.greatexpectations.io/en/latest/guides/how_to_guides/creating_batches/how_to_load_a_spark_dataframe_as_a_batch.html

runtime_batch_request = RuntimeBatchRequest(
datasource_name=“my_spark_datasource”,
data_connector_name=“my_runtime_data_connector”,
data_asset_name=“insert_your_data_asset_name_here”,
runtime_parameters={
“batch_data”: df
},
batch_identifiers={
“some_key_maybe_pipeline_stage”: “ingestion step 1”,
“some_other_key_maybe_airflow_run_id”: “run 18”
}
)

2 Likes