No support for Spark DF in Result Format COMPLETE Mode

GX OSS Support
1

it seems that when the Data Frame is Spark Data Frame (ie NOT Pandas DF), the result format, COMPLETE mode does NOT return neither:

a) unexpected_index_query
b) unexpected_list (only partial list is returned)

therefore it can be inferred that the COMPLETE mode has NOT been implemented for the case when the DF is a Spark DF - is that correct?

and therefore it is not possible to perform Automated Data Cleansing of Spark DF with GX? Only Automated Data Quality Reporting on Spark DF is possible with GZ

2

adding some new info about the issue - it seems that the bug reported above is manifested ONLY when the Result Format is specified from the Validator class e.g.:

validation_result = my_validator.expect_column_values_to_be_in_set(
column=“my_var”,
value_set=[“A”, “B”],
result_format={
“result_format”: “COMPLETE”,
“unexpected_index_column_names”: [“pk_column”],
“return_unexpected_index_query”: True,
},
)

However if the result format is specified in the Checkpoint class (and therefore applies to ALL Validators / Validation Rules referenced by the Checkpoint), the COMPLETE mode works as expected it it produces a SPark DF query which can be used to filter out bad records / records which have failed the validation rules

my investigation and explanation is that when the Validator object is persisted / saved the result format doesnt appear in the json file for some reason - which may be a bug in GX. Hence subsequently the Checkpoint referencing the Expectation (validation rule) retrieves Expectation WITHOUT result format COMLETE