# Validator.columns() returns empty list - validator.head() shows data though

**URL:** <https://discourse.greatexpectations.io/t/validator-columns-returns-empty-list-validator-head-shows-data-though/1386>\
**Category:** GX Core Support\
**Tags:** help-wanted\
**Created:** [September 13, 2023, 12:44pm UTC](https://discourse.greatexpectations.io/t/validator-columns-returns-empty-list-validator-head-shows-data-though/1386 "2023-09-13T12:44:52Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![allekai](https://avatars.discourse-cdn.com/v4/letter/a/838e76/32.png) [@allekai](https://discourse.greatexpectations.io/u/allekai)\
**Post date:** [September 13, 2023, 12:44pm UTC](https://discourse.greatexpectations.io/t/validator-columns-returns-empty-list-validator-head-shows-data-though/1386/1 "2023-09-13T12:44:52Z")

</div>

Hello everyone,

I am trying to setup GX on a table that we have stored in an S3 Storage. The connection should be done with Trino.  
I am running on Python 3.11 with GX 0.17.14 in a Jupyter Notebook.

I tried to follow the getting started guide in the documentation. Here is my code so far:

```auto
import great_expectations as gx

context = gx.get_context()

connection_string = f"trino://$USER:$PASSWORD@MY_TRINO_CONN_STRING/deltalake/deltalake_schema"

data_source = context.sources.add_sql(
    name="my_datasource", connection_string=connection_string
)
my_asset = data_source.add_query_asset(
    name="my_asset",
    query="SELECT * FROM some_table LIMIT 100",
)
my_asset.add_splitter_column_value("tenant")

my_batch_request = my_asset.build_batch_request()

context.add_or_update_expectation_suite("my_gx_suite")

validator = context.get_validator(
    batch_request=my_batch_request,
    expectation_suite_name="my_gx_suite",
)
validator.head()

```

This code shows some output:  
 ![Bild 13.09.23 um 14.29](https://canada1.discourse-cdn.com/flex031/uploads/greatexpectations/original/1X/d446bd01e15eca86d83a6259d715036d1fab42f2.jpeg)

However if I run

```auto
validator.columns()

```

it returns an empty list.

I am surprised, that I can see data in the head() output, but it seems, that somehow the data is not (really) present.

Did I forget any crucial steps?

---

<div class="post-metadata">

**Author:** ![allekai](https://avatars.discourse-cdn.com/v4/letter/a/838e76/32.png) [@allekai](https://discourse.greatexpectations.io/u/allekai)\
**Post date:** [September 13, 2023, 12:47pm UTC](https://discourse.greatexpectations.io/t/validator-columns-returns-empty-list-validator-head-shows-data-though/1386/2 "2023-09-13T12:47:04Z")

</div>

just fyi: don’t be surprised, that the returned data frame in validator.output has only a single row - our test environment sometimes has some very sparse tables 🙃

---

<div class="post-metadata">

**Author:** ![HaebichanGX](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/haebichangx/32/211_2.png) [@HaebichanGX](https://discourse.greatexpectations.io/u/HaebichanGX)\
**Post date:** [September 13, 2023, 2:26pm UTC](https://discourse.greatexpectations.io/t/validator-columns-returns-empty-list-validator-head-shows-data-though/1386/3 "2023-09-13T14:26:37Z")

</div>

Hm… @allekai that does sound very strange. Overall your code looks good, no steps forgotten. It maybe the case that your sparse tables are causing some issues that’s making it difficult for our validator to track. Please let me know if this is causing issues repeatedly. You’re also welcome to try with non sparse tables just as a testing procedure.

---

<div class="post-metadata">

**Author:** ![allekai](https://avatars.discourse-cdn.com/v4/letter/a/838e76/32.png) [@allekai](https://discourse.greatexpectations.io/u/allekai)\
**Post date:** [September 13, 2023, 2:35pm UTC](https://discourse.greatexpectations.io/t/validator-columns-returns-empty-list-validator-head-shows-data-though/1386/4 "2023-09-13T14:35:01Z")

</div>

Hi, thanks for your Feedback - I also tried with a different table, that has at least a few hundred rows. I get the same behavior with that table 😕

Here is a count(\*) grouped by tenant for the table:

![test](https://canada1.discourse-cdn.com/flex031/uploads/greatexpectations/original/1X/314220bfc21e2be6d2dee0ce88eef3d166a9efb0.png)

So the size should probably not be a problem.

I also oserve the following warning:  
EXECUTE IMMEDIATE not available for MY\_TRINO\_URL:443; defaulting to legacy prepared statements (TrinoUserError(type=USER\_ERROR, name=SYNTAX\_ERROR, message=“line 1:19: mismatched input ‘‘SELECT 1’’. Expecting: ‘USING’, ”, query\_id=20230913\_143138\_26694\_xwq25))  
when running either the `validator.head()` or `validator.expect_column_values_to_not_be_null(column="my_column")`

Interestingly the `validator.expect_column_to_exist(column="my_column")` runs without warning (but of course with the value `"sucess": false`, which makes sense, since the validator doesn’t find any columns

---

<div class="post-metadata">

**Author:** ![allekai](https://avatars.discourse-cdn.com/v4/letter/a/838e76/32.png) [@allekai](https://discourse.greatexpectations.io/u/allekai)\
**Post date:** [September 13, 2023, 2:39pm UTC](https://discourse.greatexpectations.io/t/validator-columns-returns-empty-list-validator-head-shows-data-though/1386/5 "2023-09-13T14:39:35Z")

</div>

Here is my stacktrace when running `validator.expect_column_values_to_not_be_null(column="my_column")`:  
Somewhere in the MetricsCalculator part are lost? I am new to GX, still trying to wrap my head around the concepts 😅

```auto
InvalidMetricAccessorDomainKwargsKeyError Traceback (most recent call last)
File ~/opt/anaconda3/envs/gx/lib/python3.11/site-packages/great_expectations/execution_engine/execution_engine.py:548, in ExecutionEngine._process_direct_and_bundled_metric_computation_configurations(self, metric_fn_direct_configurations, metric_fn_bundle_configurations)
    545 try:
    546 resolved_metrics[
    547 metric_computation_configuration.metric_configuration.id
--> 548 ] = metric_computation_configuration.metric_fn( # type: ignore[misc] # F not callable
    549 **metric_computation_configuration.metric_provider_kwargs
    550 )
    551 except Exception as e:

File ~/opt/anaconda3/envs/gx/lib/python3.11/site-packages/great_expectations/expectations/metrics/metric_provider.py:90, in metric_partial.<locals>.wrapper.<locals>.inner_func(*args, **kwargs)
     88 @wraps(metric_fn)
     89 def inner_func(*args, **kwargs):
---> 90 return metric_fn(*args, **kwargs)

File ~/opt/anaconda3/envs/gx/lib/python3.11/site-packages/great_expectations/expectations/metrics/map_metric_provider/column_condition_partial.py:169, in column_condition_partial.<locals>.wrapper.<locals>.inner_func(cls, execution_engine, metric_domain_kwargs, metric_value_kwargs, metrics, runtime_configuration)
    154 @metric_partial(
    155 engine=engine,
    156 partial_fn_type=partial_fn_type,
   (...)
    167 runtime_configuration: dict,
    168 ):
--> 169 metric_domain_kwargs = get_dbms_compatible_metric_domain_kwargs(
    170 metric_domain_kwargs=metric_domain_kwargs,
    171 batch_columns_list=metrics["table.columns"],
    172 )
    174 (
    175 selectable,
    176 compute_domain_kwargs,
   (...)
    179 domain_kwargs=metric_domain_kwargs, domain_type=domain_type
    180 )

File ~/opt/anaconda3/envs/gx/lib/python3.11/site-packages/great_expectations/expectations/metrics/util.py:629, in get_dbms_compatible_metric_domain_kwargs(metric_domain_kwargs, batch_columns_list)
    628 if "column" in metric_domain_kwargs:
--> 629 column_name: str | sqlalchemy.quoted_name = get_dbms_compatible_column_names(
    630 column_names=metric_domain_kwargs["column"],
    631 batch_columns_list=batch_columns_list,
    632 )
    633 metric_domain_kwargs["column"] = column_name

File ~/opt/anaconda3/envs/gx/lib/python3.11/site-packages/great_expectations/expectations/metrics/util.py:703, in get_dbms_compatible_column_names(column_names, batch_columns_list, error_message_template)
    683 """
    684 Case non-sensitivity is expressed in upper case by common DBMS backends and in lower case by SQLAlchemy, with any
    685 deviations enclosed with double quotes.
   (...)
    698 Single property-typed column name object or list of property-typed column name objects (depending on input).
    699 """
    700 normalized_typed_batch_columns_mappings: List[
    701 Tuple[str, str | sqlalchemy.quoted_name]
    702 ] = (
--> 703 _verify_column_names_exist_and_get_normalized_typed_column_names_map(
    704 column_names=column_names,
    705 batch_columns_list=batch_columns_list,
    706 error_message_template=error_message_template,
    707 )
    708 or []
    709 )
    711 element: Tuple[str, str | sqlalchemy.quoted_name]

File ~/opt/anaconda3/envs/gx/lib/python3.11/site-packages/great_expectations/expectations/metrics/util.py:799, in _verify_column_names_exist_and_get_normalized_typed_column_names_map(column_names, batch_columns_list, error_message_template, verify_only)
    798 if normalized_column_name_mapping is None:
--> 799 raise gx_exceptions.InvalidMetricAccessorDomainKwargsKeyError(
    800 message=error_message_template.format(column_name=column_name)
    801 )
    802 else: # noqa: PLR5501

InvalidMetricAccessorDomainKwargsKeyError: Error: The column "my_column" in BatchData does not exist.

The above exception was the direct cause of the following exception:

MetricResolutionError Traceback (most recent call last)
Cell In[43], line 2
      1 import pdb; pdb.set_trace()
----> 2 validator.expect_column_values_to_not_be_null(column="my_column")

File ~/opt/anaconda3/envs/gx/lib/python3.11/site-packages/great_expectations/validator/validator.py:594, in Validator.validate_expectation.<locals>.inst_expectation(*args, **kwargs)
    588 validation_result = ExpectationValidationResult(
    589 success=False,
    590 exception_info=exception_info,
    591 expectation_config=configuration,
    592 )
    593 else:
--> 594 raise err
    596 if self._include_rendered_content:
    597 validation_result.render()

File ~/opt/anaconda3/envs/gx/lib/python3.11/site-packages/great_expectations/validator/validator.py:557, in Validator.validate_expectation.<locals>.inst_expectation(*args, **kwargs)
    553 validation_result = ExpectationValidationResult(
    554 expectation_config=copy.deepcopy(expectation.configuration)
    555 )
    556 else:
--> 557 validation_result = expectation.validate(
    558 validator=self,
    559 evaluation_parameters=self._expectation_suite.evaluation_parameters,
    560 data_context=self._data_context,
    561 runtime_configuration=basic_runtime_configuration,
    562 )
    564 # If validate has set active_validation to true, then we do not save the config to avoid
    565 # saving updating expectation configs to the same suite during validation runs
    566 if self._active_validation is True:

File ~/opt/anaconda3/envs/gx/lib/python3.11/site-packages/great_expectations/expectations/expectation.py:1276, in Expectation.validate(self, validator, configuration, evaluation_parameters, interactive_evaluation, data_context, runtime_configuration)
   1267 self._warn_if_result_format_config_in_expectation_configuration(
   1268 configuration=configuration
   1269 )
   1271 configuration.process_evaluation_parameters(
   1272 evaluation_parameters, interactive_evaluation, data_context
   1273 )
   1274 expectation_validation_result_list: list[
   1275 ExpectationValidationResult
-> 1276 ] = validator.graph_validate(
   1277 configurations=[configuration],
   1278 runtime_configuration=runtime_configuration,
   1279 )
   1280 return expectation_validation_result_list[0]

File ~/opt/anaconda3/envs/gx/lib/python3.11/site-packages/great_expectations/validator/validator.py:1069, in Validator.graph_validate(self, configurations, runtime_configuration)
   1067 return evrs
   1068 else:
-> 1069 raise err
   1071 configuration: ExpectationConfiguration
   1072 result: ExpectationValidationResult

File ~/opt/anaconda3/envs/gx/lib/python3.11/site-packages/great_expectations/validator/validator.py:1048, in Validator.graph_validate(self, configurations, runtime_configuration)
   1041 resolved_metrics: _MetricsDict
   1043 try:
   1044 (
   1045 resolved_metrics,
   1046 evrs,
   1047 processed_configurations,
-> 1048 ) = self._resolve_suite_level_graph_and_process_metric_evaluation_errors(
   1049 graph=graph,
   1050 runtime_configuration=runtime_configuration,
   1051 expectation_validation_graphs=expectation_validation_graphs,
   1052 evrs=evrs,
   1053 processed_configurations=processed_configurations,
   1054 show_progress_bars=self._determine_progress_bars(),
   1055 )
   1056 except Exception as err:
   1057 # If a general Exception occurs during the execution of "ValidationGraph.resolve()", then
   1058 # all expectations in the suite are impacted, because it is impossible to attribute the failure to a metric.
   1059 if catch_exceptions:

File ~/opt/anaconda3/envs/gx/lib/python3.11/site-packages/great_expectations/validator/validator.py:1207, in Validator._resolve_suite_level_graph_and_process_metric_evaluation_errors(self, graph, runtime_configuration, expectation_validation_graphs, evrs, processed_configurations, show_progress_bars)
   1199 resolved_metrics: _MetricsDict
   1200 aborted_metrics_info: Dict[
   1201 _MetricKey,
   1202 Dict[str, Union[MetricConfiguration, Set[ExceptionInfo], int]],
   1203 ]
   1204 (
   1205 resolved_metrics,
   1206 aborted_metrics_info,
-> 1207 ) = self._metrics_calculator.resolve_validation_graph(
   1208 graph=graph,
   1209 runtime_configuration=runtime_configuration,
   1210 min_graph_edges_pbar_enable=0,
   1211 )
   1213 # Trace MetricResolutionError occurrences to expectations relying on corresponding malfunctioning metrics.
   1214 rejected_configurations: List[ExpectationConfiguration] = []

File ~/opt/anaconda3/envs/gx/lib/python3.11/site-packages/great_expectations/validator/metrics_calculator.py:287, in MetricsCalculator.resolve_validation_graph(self, graph, runtime_configuration, min_graph_edges_pbar_enable)
    282 resolved_metrics: _MetricsDict
    283 aborted_metrics_info: Dict[
    284 _MetricKey,
    285 Dict[str, Union[MetricConfiguration, Set[ExceptionInfo], int]],
    286 ]
--> 287 resolved_metrics, aborted_metrics_info = graph.resolve(
    288 runtime_configuration=runtime_configuration,
    289 min_graph_edges_pbar_enable=min_graph_edges_pbar_enable,
    290 show_progress_bars=self._show_progress_bars,
    291 )
    292 return resolved_metrics, aborted_metrics_info

File ~/opt/anaconda3/envs/gx/lib/python3.11/site-packages/great_expectations/validator/validation_graph.py:209, in ValidationGraph.resolve(self, runtime_configuration, min_graph_edges_pbar_enable, show_progress_bars)
    203 resolved_metrics: Dict[_MetricKey, MetricValue] = {}
    205 # updates graph with aborted metrics
    206 aborted_metrics_info: Dict[
    207 _MetricKey,
    208 Dict[str, Union[MetricConfiguration, Set[ExceptionInfo], int]],
--> 209 ] = self._resolve(
    210 metrics=resolved_metrics,
    211 runtime_configuration=runtime_configuration,
    212 min_graph_edges_pbar_enable=min_graph_edges_pbar_enable,
    213 show_progress_bars=show_progress_bars,
    214 )
    216 return resolved_metrics, aborted_metrics_info

File ~/opt/anaconda3/envs/gx/lib/python3.11/site-packages/great_expectations/validator/validation_graph.py:315, in ValidationGraph._resolve(self, metrics, runtime_configuration, min_graph_edges_pbar_enable, show_progress_bars)
    311 failed_metric_info[failed_metric.id]["exception_info"] = {
    312 exception_info
    313 }
    314 else:
--> 315 raise err
    316 except Exception as e:
    317 if catch_exceptions:

File ~/opt/anaconda3/envs/gx/lib/python3.11/site-packages/great_expectations/validator/validation_graph.py:285, in ValidationGraph._resolve(self, metrics, runtime_configuration, min_graph_edges_pbar_enable, show_progress_bars)
    280 computable_metrics.add(metric)
    282 try:
    283 # Access "ExecutionEngine.resolve_metrics()" method, to resolve missing "MetricConfiguration" objects.
    284 metrics.update(
--> 285 self._execution_engine.resolve_metrics(
    286 metrics_to_resolve=computable_metrics, # type: ignore[arg-type] # Metric typing needs further refinement.
    287 metrics=metrics, # type: ignore[arg-type] # Metric typing needs further refinement.
    288 runtime_configuration=runtime_configuration,
    289 )
    290 )
    291 progress_bar.update(len(computable_metrics))
    292 progress_bar.refresh()

File ~/opt/anaconda3/envs/gx/lib/python3.11/site-packages/great_expectations/execution_engine/execution_engine.py:283, in ExecutionEngine.resolve_metrics(self, metrics_to_resolve, metrics, runtime_configuration)
    274 metric_fn_bundle_configurations: List[MetricComputationConfiguration]
    275 (
    276 metric_fn_direct_configurations,
    277 metric_fn_bundle_configurations,
   (...)
    281 runtime_configuration=runtime_configuration,
    282 )
--> 283 return self._process_direct_and_bundled_metric_computation_configurations(
    284 metric_fn_direct_configurations=metric_fn_direct_configurations,
    285 metric_fn_bundle_configurations=metric_fn_bundle_configurations,
    286 )

File ~/opt/anaconda3/envs/gx/lib/python3.11/site-packages/great_expectations/execution_engine/execution_engine.py:552, in ExecutionEngine._process_direct_and_bundled_metric_computation_configurations(self, metric_fn_direct_configurations, metric_fn_bundle_configurations)
    546 resolved_metrics[
    547 metric_computation_configuration.metric_configuration.id
    548 ] = metric_computation_configuration.metric_fn( # type: ignore[misc] # F not callable
    549 **metric_computation_configuration.metric_provider_kwargs
    550 )
    551 except Exception as e:
--> 552 raise gx_exceptions.MetricResolutionError(
    553 message=str(e),
    554 failed_metrics=(
    555 metric_computation_configuration.metric_configuration,
    556 ),
    557 ) from e
    559 try:
    560 # an engine-specific way of computing metrics together
    561 resolved_metric_bundle: Dict[
    562 Tuple[str, str, str], MetricValue
    563 ] = self.resolve_metric_bundle(
    564 metric_fn_bundle=metric_fn_bundle_configurations
    565 )

MetricResolutionError: Error: The column "my_column" in BatchData does not exist.

```

---

<div class="post-metadata">

**Author:** ![allekai](https://avatars.discourse-cdn.com/v4/letter/a/838e76/32.png) [@allekai](https://discourse.greatexpectations.io/u/allekai)\
**Post date:** [September 14, 2023, 7:04am UTC](https://discourse.greatexpectations.io/t/validator-columns-returns-empty-list-validator-head-shows-data-though/1386/6 "2023-09-14T07:04:33Z")

</div>

I will try it again with an older version of GX - I hope I can find time today

---

<div class="post-metadata">

**Author:** ![allekai](https://avatars.discourse-cdn.com/v4/letter/a/838e76/32.png) [@allekai](https://discourse.greatexpectations.io/u/allekai)\
**Post date:** [September 18, 2023, 6:28am UTC](https://discourse.greatexpectations.io/t/validator-columns-returns-empty-list-validator-head-shows-data-though/1386/7 "2023-09-18T06:28:34Z")

</div>

Unfortunately, I still get an error even with the downgraded Version 0.16.16

@HaebichanGX should I open an GitHub issue for this? Or how can we proceed here?

---

<div class="post-metadata">

**Author:** ![HaebichanGX](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/haebichangx/32/211_2.png) [@HaebichanGX](https://discourse.greatexpectations.io/u/HaebichanGX)\
**Post date:** [October 2, 2023, 6:37pm UTC](https://discourse.greatexpectations.io/t/validator-columns-returns-empty-list-validator-head-shows-data-though/1386/8 "2023-10-02T18:37:14Z")

</div>

Hey @allekai pardon the late response. I don’t seem to get notified well on these. Yes please create a Github issue for these.

---

<div class="post-metadata">

**Author:** ![Freya](https://avatars.discourse-cdn.com/v4/letter/f/a9a28c/32.png) [@Freya](https://discourse.greatexpectations.io/u/Freya)\
**Post date:** [January 22, 2024, 1:12am UTC](https://discourse.greatexpectations.io/t/validator-columns-returns-empty-list-validator-head-shows-data-though/1386/9 "2024-01-22T01:12:28Z")

</div>

hey @HaebichanGX is this issue solved? seems I faced same problem [The column "xxx" in BatchData does not exist for query asset batch request](https://discourse.greatexpectations.io/t/the-column-xxx-in-batchdata-does-not-exist-for-query-asset-batch-request/1589)

---

<div class="post-metadata">

**Author:** ![HaebichanGX](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/haebichangx/32/211_2.png) [@HaebichanGX](https://discourse.greatexpectations.io/u/HaebichanGX)\
**Post date:** [January 24, 2024, 3:48pm UTC](https://discourse.greatexpectations.io/t/validator-columns-returns-empty-list-validator-head-shows-data-though/1386/10 "2024-01-24T15:48:59Z")

</div>

Hi @Freya we’re on our way to release GX 1.0, our biggest overhaul for the OSS version, so there unfortunately won’t be much development in our existing versions. Let me take a look at this issue in case there are additional information on this.

Also in your situation, can you please try `add_table_asset` instead of query asset to see if that resolves the issue for now?