Hello all, this is me, Vladimir. I’ve had this post hanging in Slack for ± 20 days with no response, so I am cross posting it here, in hopes that it will get attention.
Quite frequently in my data engineering work, I run into a scenario where I need to write a customized SQL check where SQL code references a variable column name. In the official documentation, such scenarios are unforeseen, as the quickstart only references the {batch} template. Therefore, I wrote my own abstract base class for scenarios like these, on the basis of a brief overlook of some cases in the experimental folder in GX Core source code:
I would really like to upstream this code, as no doubt such a scenario is something other people would run into all the time. However, I am not sure where to start. Would I need to make some changes to my already existing code? Would you like me to write official documentation for it as well? Where exactly would this bit of code belong in your code structure? I am more than willing to do everything required here. Please let me know what the next steps for me are to take before I submit a pull request.
Hey @VovaVili, thanks for reaching out. We would be happy to accept a PR for this Expectation. I took a quick look at the gist you put together, and here’s some early feedback:
This Expectation should live in contrib/experimental/great_expectations_experimental/expectations
I would suggest using the UnexpectedRowsExpectation as a base class rather than the QueryExpectation. We’ve removed QueryExpectation from our docs, and the UnexpectedRowsExpectation is the preferred way to make tests from arbitrary SQL in GX v1.
Add test coverage in /tests/integration/contrib using the parametrize_batch_for_data_sources pattern. As an example, see any test module in the tests/integration/data_sources_and_expectations/expectations package.
We won’t have capacity to help out with the implementation, but will prioritize reviewing and merging the PR once its ready. Thanks!