When I try to use the below GX code, I am getting the below error. I am using GX using pyspark and databricks. When I try to comment the row condition statements in the expectaton, then we are not seeing the error. Can you please help to resolve.
Error: [CANNOT_RESOLVE_DATAFRAME_COLUMN] Cannot resolve dataframe column “colA”. It’s probably because of illegal references like df1.select(df2.col("a")). SQLSTATE: 42704
When I remove the row_condition as below, error on colA does not appear
Code:
expectation_with_condition = gx.expectations.ExpectColumnValuesToNotBeNull(
column=“colA”,
)
Hello! Same happens to me, but with different expectation:
gxe.ExpectColumnMinToBeBetween(
column=“len_not_already_purchased_recommendations”,
min_value=50,
max_value=50,
condition_parser=“spark”, row_condition=‘col(“parameters.algorithm”).notNull()’),
ERROR:
“exception_message”: “[CANNOT_RESOLVE_DATAFRAME_COLUMN] Cannot resolve dataframe column "len_family_activation_recommendations". It’s probably because of illegal references like df1.select(df2.col(\"a\")). SQLSTATE: 42704”,
I don’t find GX 1.0 doc on this but in the older version 0.18.x , for spark the condition_parser should be “great_expectations__experimental__”. I’m wondering if this has changed for GX 1.0?