Expect_compound_columns_to_be_unique got an unexpected keyword argument 'condition_parser'

GX OSS Support
1

I have created a GE dataset using SparkDFDataset(sparkDF) and tried to add expectation on it as df.expect_compound_columns_to_be_unique(
columns, condition_parser=‘great_expectations__experimental__’, row_condition=‘col(“properties_eye”)==“black”’), It was throwing the error expect_compound_columns_to_be_unique() got an unexpected keyword argument ‘condition_parser’, it was working fine for pandas.

How to use row_condition and condition_parser for sparkDFDataset?

2

Hi @vishwanath560, unfortunately, condition_parser is not included with SparkDFDataset. In fact, the Dataset construct is now deprecated, so you may find that row_condition behaves in unexpected ways. I would suggest taking a look at the Spark implementation for in-memory source data as an alternative. A guide on how to do so can be found in our docs.