No support for Spark DF in Result Format COMPLETE Mode

it seems that when the Data Frame is Spark Data Frame (ie NOT Pandas DF), the result format, COMPLETE mode does NOT return neither:

a) unexpected_index_query
b) unexpected_list (only partial list is returned)

therefore it can be inferred that the COMPLETE mode has NOT been implemented for the case when the DF is a Spark DF - is that correct?

and therefore it is not possible to perform Automated Data Cleansing of Spark DF with GX? Only Automated Data Quality Reporting on Spark DF is possible with GZ

adding some new info about the issue - it seems that the bug reported above is manifested ONLY when the Result Format is specified from the Validator class e.g.:

validation_result = my_validator.expect_column_values_to_be_in_set(
value_set=[“A”, “B”],
“result_format”: “COMPLETE”,
“unexpected_index_column_names”: [“pk_column”],
“return_unexpected_index_query”: True,

However if the result format is specified in the Checkpoint class (and therefore applies to ALL Validators / Validation Rules referenced by the Checkpoint), the COMPLETE mode works as expected it it produces a SPark DF query which can be used to filter out bad records / records which have failed the validation rules

my investigation and explanation is that when the Validator object is persisted / saved the result format doesnt appear in the json file for some reason - which may be a bug in GX. Hence subsequently the Checkpoint referencing the Expectation (validation rule) retrieves Expectation WITHOUT result format COMLETE