It works with Sybase and Sqlalchemy and is testing great expectations using the CLI.
Problem)
- If you use more than one expectation rule for one suit.json, there seems to be a query that overlaps according to the rule.
ex) I only derived the column_values_to_be_in_set condition, but the issue of not null verification query running at the same time - Problems that cause errors due to overlapping Alias’ names in the query used at this time
Therefore, I would like to think about a solution together.
gx version : 0.15.50
sqlalchemy version : 1.4.48
Example)
{
"data_asset_type":null,
"expectations_suite_name : "test_suite.json"
"expectations" : [
{
"expectation_type" : "expect_column_values_to_be_in_set"
"kwargs": {
"column" : "col1"
"value_set" : [0,1]
},
"meta":{}
},
{
"expectation_type" : "expect_column_values_to_be_between"
"kwargs": {
"column" : "col2"
"min_value" : 1,
"max_value": 99
},
"meta":{}
}
]
}
error)
SELECT sum((CASE WHEN ("col1" IS NOT NULL AND "col1" IS NOT IN (0,1))) THEN 1 ELSE 0 END) AS "column_values.in_set.unexpected_count",
sum(CASE WHEN ("col2" IS NOT NULL AND NOT "col2" >=1 and AND "co2" <=99)) THEN 1 ELSE 0 END) AS "column_values_between.unexpected_count
, sum(CASE WHEN("col2" IS NULL) THEN 1 ELSE 0 END) AS "column_values.nonnull.unexpected_count"
, sum(CASE WHEN("col1" IS NULL) THEN 1 ELSE 0 END) AS "column_values.nonnull.unexpected_count FROM (SELECT * FROM TABLENAME) AS anon_1)
As we now know, we validate the data through sqlalchemy’s functions (in_,or_,etc) and we realize that they also act as python codes. I wonder where the query runs and if there is no problem changing the alias.