Hi all,
I have been observing an abnormal bahavior on this expectation on the GX 0.18.8 on Spark, Databricks.
Basically what is happening is that it is processing around 1.6Million rows and failing only 11 records in most cases, however, it is providing the unexpected_count = 0 and unexpected_index_list = .
Other times, it is still providing unexpected_count = 0, but unexpected_index_list comes filled with values.
Moreover, it is marking the status of the expectation as success, sample below.
Do you have any idea on what might be happening? Can it be due to very low number of failures compared to the input data and some approximations are happening?
It is also crazy that this expectation is working properly on other use cases, so any tip would be appreciated.
Note: I am not providing the mostly parameter, so it should fail even if a single record is failing, as the default is 100%.
Thanks in advance.
Best Regards,
José Correia