How to run an expectation only if the column exists

Archive
1

Hi Team, Hi Community,

I’ve been trying to get an expectation to work but I think I need help. I want to use expect_column_values_to_be_between, but only actually run the expectation if the column exists. If the column does not exist, I do not care about validating it. My data source is a CSV with pandas.

When just coding the expectation, I get KeyError on files where the given column does not exist and passing in catch_exceptions catches the exceptions but still fails the suite and marks the expectation as error. I want it to succeed if the column doesn’t exist and validate if the column does exist. I’ve been trying to get row_condition to work for me, but the pandas query engine doesn’t seem to support querying whether a column exists as it is intended for expression syntax in columns.

1 Like
2

Currently Expectations do not support this conditionality (“if column exists”). This might be a good enhancement. For now, one way to solve it is to create two Expectation Suites - one with this expectations and one without. Then you can decide which one to apply to a particular batch of data, depending on the presence of the column.

1 Like
3

Hi, any idea in how can I proceed if I need this use case? I mean, for me is unfeasible to build several suites because I don’t know when a column won’t be present. I need to validate a column only if it exists on the data set.

4

I’m also very keen to see this feature. Having one uber-suite would be much more manageable for us compared to having lots and lots of suites.