Unable to parse condition: col("guid").isNull()

GX Core Support
1

I’m running GX in a Spark notebook in Databricks. The data source is a Dataframe read from a Delta table. The code for adding conditional expectation is like below:
“”"

Conditional Contract

ml_suite.add_expectation(
gx.expectations.ExpectColumnValuesToBeNull(column=‘zuid’,
condition_parser=‘great_expectations__experimental__’,
row_condition=‘col(“guid”).isNull()’)
)
“”"

However, i ran into error like
“”"
raise gx_exceptions.MetricResolutionError(\ngreat_expectations.exceptions.exceptions.MetricResolutionError: unable to parse condition: col("guid").isNull()\n",
“exception_message”: “unable to parse condition: col("guid").isNull()”,
“”"

How should I get rid of this?

2

can you try changing the parser to “spark” and then updating the row condition to use standard SQL syntax:

here’s a snippet from a working example that i have with a spark/databricks:

from datetime import datetime
import great_expectations as gx

cond_expectation = gx.expectations.ExpectColumnValuesToBeInSet(
    column="tpep_dropoff_datetime",
    value_set=[
        datetime(2016, 1, 1, 0, 12),
        datetime(2016, 1, 1, 0, 13),
    ],
    condition_parser="spark",
    row_condition='`pickup_zip` = 10001'  
)

batch.validate(cond_expectation)

pickup_zip is my column name that i have wrapped in backticks

3

Yes that works! Thanks!

1 Like