Create customize expectation and use it in Airflow DAG

Archive
1

Hi everyone,
It is possible to create a simple suite with only one expectation (let’s say it expect_column_values_to_not_be_null ) and then call the GreatExpectation Operator and send the column name as variable/parameter?

1 Like
2

@shaharp Currently the name of the column to which an column expectation applies must be provided at the time of adding the expectation to the suite. One of the reasons is that this argument is part of the identity of the expectation and is used by the logic that determines if two expectations in a suite are duplicates.

Could you please describe your use case in a bit more details, so that we can discuss how we can address it? Specifically, why you can’t add the expectation to all the columns you want to check upfront?

3

@eugene.mandel
We have a lot of Airflow users who do not want to “mess” with writing expectations.
We want to provide them with ready-made expectations and that they will only have to enter the parameter they want.

Another use-case is that we want to set a different expectaion value per running environment- for example, if the expectation_type is “expect_table_row_count_to_be_between” I want to set different values for the “min_value” key (dev env - 10 rows, prod env - 1000 rows).