Reusing already existing checks

Feedback
1

Does GE have some possibility of reusing already executed checks? We are using Big Query and it might be expensive to scan tables on each check.

The idea here would be either to store the query output somewhere or create a view that lives for a certain period of time so it can by default be reused by a check using the same or similar parameters in the future.

Due to size of the datasets in the company I work at, the amount of the data scans will increase with the amount of queries. This will lead to duplication and significant cost increase.

We solved it for now by developing custom operators that load check outputs to a postgredb

2 Likes
2

Great Expectations’ [Evaluation Parameters] (Evaluation Parameters — great_expectations documentation) feature allows an Expectation to reference another Expectation’s Validation Result. A lot of use cases can be implemented on top of this feature.
A more general support for this will be Multi Batch Expectations that will be released in the future. They will allow to express tests like “I expect the median of a column in this batch to be within 5% of the median in the last 10 batches”.