Checkpoint results to delta lake

GX Core Support
1

Hi Team,
I am using GX version 1.0.2 in a Microsoft Fabric notebook and I am able to create a checkpoint and display the results. The real challenge comes in converting the checkpoint results to a normalized json format. I do not find any documentation regarding the same in your website.
Below are the steps done. Please go to the code at the bottom and advise on creating a normalized json from checkpoint results.
1.Created a Context
2.Created data_source,data_asset,batch_definition
3.Created a suite
3.1 Created expectations and added to suite
4.Created batch_parameters,batch and validation_results
5.Created validation_definition, action_list
6.Created checkpoint and added it to context and executed it
7. I am able to display results

runid = gx.RunIdentifier(run_name=“my_checkpoint”)
results = checkpoint.run(batch_parameters=batch_parameters, run_id=runid)

How can I convert the results to a normalized json format and read it via a pyspark dataframe.

2 Likes
2

I guess this is the first step:
chk_dict = list(validation_results.run_results.values())[0].to_json_dict()

Them you need to write a program to normalize the returned JSON similar the one explained at Tutorial: Validate data using SemPy and Great Expectations (GX) - Microsoft Fabric | Microsoft Learn

1 Like