[PARSE_SYNTAX_ERROR] for ExpectColumnValuesToBeUnique when column name has space in it (PySpark)

GX Core Support
1

GX Version = 1.23.2

Expectation = ExpectColumnValuesToBeUnique

Data Source = Pyspark (context.data_sources.add_spark(name=data_source_name))

Asset = add_dataframe_asset

Column = “Incident Number”

Error = [PARSE_SYNTAX_ERROR] Syntax error at or near ‘IS’. SQLSTATE: 42601 (line 1, pos 16)

**
expect_column_values_to_be_unique raised an exception:
[PARSE_SYNTAX_ERROR] Syntax error at or near 'IS'. SQLSTATE: 42601 (line 1, pos 16) == SQL == Incident Number IS NOT NULL ----------------^^^**

Show exception traceback…

Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/great_expectations/execution_engine/execution_engine.py", line 577, in _process_direct_and_bundled_metric_computation_configurations
metric_computation_configuration.metric_fn( # type: ignore[misc] # F not callable
File "/usr/local/lib/python3.12/dist-packages/great_expectations/expectations/metrics/map_metric_provider/map_condition_auxilliary_methods.py", line 633, in _spark_map_condition_unexpected_count_value
df = execution_engine.get_domain_records(domain_kwargs=domain_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/great_expectations/execution_engine/sparkdf_execution_engine.py", line 740, in get_domain_records
data = data.filter(filter_condition.condition)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/pyspark/sql/classic/dataframe.py", line 1010, in filter
jdf = self._jdf.filter(condition)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/py4j/java_gateway.py", line 1362, in __call__
return_value = get_return_value(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/pyspark/errors/exceptions/captured.py", line 288, in deco
raise converted from None
pyspark.errors.exceptions.captured.ParseException:
[PARSE_SYNTAX_ERROR] Syntax error at or near 'IS'. SQLSTATE: 42601 (line 1, pos 16)

== SQL ==
Incident Number IS NOT NULL
----------------^^^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/great_expectations/validator/validation_graph.py", line 290, in _resolve
self._execution_engine.resolve_metrics(
File "/usr/local/lib/python3.12/dist-packages/great_expectations/execution_engine/execution_engine.py", line 294, in resolve_metrics
return self._process_direct_and_bundled_metric_computation_configurations(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/great_expectations/execution_engine/execution_engine.py", line 582, in _process_direct_and_bundled_metric_computation_configurations
raise gx_exceptions.MetricResolutionError(
great_expectations.exceptions.exceptions.MetricResolutionError:
[PARSE_SYNTAX_ERROR] Syntax error at or near 'IS'. SQLSTATE: 42601 (line 1, pos 16)

== SQL ==
Incident Number IS NOT NULL
----------------^^^

Note: When the space is removed from the column name this expectation works fine

image
image1354×333 10.7 KB

2

Hi @KK1259, thanks for the bug report. This is a GX bug - i’ve filed an issue to track the fix. As a workaround until that lands, you can rename your column to a plain identifier prior to validation:

 df = df.withColumnRenamed("Incident Number", "Incident_Number")

The bug also affects identifiers with -. Thanks again for reporting this bug!