Abnormal behavior on expectation expect_column_pair_values_a_to_be_greater_than_b

GX Core Support
1

Hi all,

I have been observing an abnormal bahavior on this expectation on the GX 0.18.8 on Spark, Databricks.
Basically what is happening is that it is processing around 1.6Million rows and failing only 11 records in most cases, however, it is providing the unexpected_count = 0 and unexpected_index_list = .
Other times, it is still providing unexpected_count = 0, but unexpected_index_list comes filled with values.
Moreover, it is marking the status of the expectation as success, sample below.

image
image1494×410 61 KB

Do you have any idea on what might be happening? Can it be due to very low number of failures compared to the input data and some approximations are happening?
It is also crazy that this expectation is working properly on other use cases, so any tip would be appreciated.
Note: I am not providing the mostly parameter, so it should fail even if a single record is failing, as the default is 100%.

Thanks in advance.

Best Regards,
José Correia

2

unfortunately we no longer offer support for version 0.18.8 but we’d be happy to take a look if this behavior is still present in latest versions. I’d recommend upgrading at your earliest convenience and if this is still an issue in that case, please let us know