How to add mulitple checks in single expectation to reduce cost (not in expectation_suite)

Archive
, , ,
1

We are trying to achieve a complex rule check like :

“COL_A>COL_B and COL_C IS NOT NULL and (COL_D in IN (SELECT COL_E from another table where main_table.key=another_table.key) OR COL_D is NULL)*”.

There are some points which we are struggling to sort out :

  1. How to add multiple checks in one expectation suite so that we can reduce the query cost. Our datasets are hosted in cloud and is pretty costly to query multiple times. How can we ask the sqlalchemy backend of the GE to combine all expectations in one and then run them in once. We could find out the way of sending the sa.filter to achieve similar thing , but in that case we need to rewrite all the rules, even though there might be existing expectations that can do some blocks of the checks like : COL_A>COL_B , COL_C IS NOT NULL etc and it would be great if all the required checks could be appended with required ‘sa.and_/sa.or_’ .

  2. How can we check something like "COL_D in IN (SELECT COL_E from another table where main_table.key=another_table.key) ", the nearest we have , takes a static value set, but how to have that part also dynamic for scenarios like this ?

  3. A clear documentation would always help to build customized expectations quickly, probably something that shows how the parameter flows from expectation to the execution engine , with the explanation of the classes touched in between. I know its lot to ask for :slight_smile: