Create Custom Expectations using pandas dataframe in ingestion layer

Archive
1

We have to create some data quality checks which involves multiple columns of the input file.
For this check we have to sort the data fields first and then group by the data and apply some lead lag functions to produce the error record.

We are able to perform this function in python pandas dataframe but we are not sure as how to build this rule in great expectations tool in custom expectation.

Is this possible to create such kind of check at data ingestion layer using 4 to 5 columns and apply such functions in pandas dataframe.

1 Like
2

My suggestion is to create a new data asset from your original file by applying all your transforms to get a data frame that can be validated using the standard expectations (instead of implementing a complex custom expectation.

1 Like