Hi GX community! I am using 0.18.3 and query asset to get batch request and do data validation. But there is always error saying “The column “xxx” in BatchData does not exist.”
When I print validator.head() I could see the column, however once add validate rule e.g. validator.expect_column_values_to_not_be_null(column=“my_column”) then it will throw out error no such column “The column “my_column” in BatchData does not exist.” my code is like this:
query_name = "select * from my_table"
trino_datasource.add_query_asset(name=asset_name, query=query_name)
my_asset = data_context.get_datasource("trino").get_asset(asset_name)
batch_request = my_asset.build_batch_request()
expectation_suite_name = "test_column_not_null"
data_context.add_or_update_expectation_suite(expectation_suite_name=expectation_suite_name)
validator = data_context.get_validator(batch_request=batch_request, expectation_suite_name=expectation_suite_name)
validator.expect_column_values_to_not_be_null(column="my_column") (edited)
Hey @Freya just out of curiosity, does add_table_asset work? I believe this issue doesn’t occur on this method. And since you’re pulling the whole table anyway.
hi @HaebichanGX ，yes “add_table_asset” works perfectly. But now we need to do some data filter, so I need to use add_query_asset.
or is there any workaround we could filter data?
Note: @Wanyun Ding originally posted this reply in Slack. It might not have transferred perfectly.