Get GX to output SQL and wait for a response

GX Core Support
1

Hi,

I’m currently trying to implement Great Expectations on some infrastructure that executes SQL queries against data in a non-standard way. As a result, none of the built-in fluent data sources work with this infrastructure.

What would be ideal for me is if there was a way (maybe by creating a custom data source) to get GX to generate the SQL for a given expectation (or set of expectations), then pass it to some function I define, then that function responds with the query result, at which point GX proceeds as normal.

Is there an easy way to do this? If not, what would be the least hard way to do this?

Thanks

2

Hey @lcoogan - interesting question. There isn’t a way to do this using our public API. The least hard way to do this might be to implement a custom version of the SqlAlchemyExecutionEngine. GX doesn’t support custom ExecutionEngines so you would need to maintain a fork, and I would expect the implementation to be complex.

If you decide to try an implementation, you could either override all the methods on the execution engine that make an external call with your custom logic, or you could intercept the underlying SqlAlchemy engine and add custom event listeners. Unfortunately i think both solutions would be likely to encounter unexpected issues, and would be a hassle for you to maintain.

3

Thank you for the very fast and clear response Josh, I appreciate it.

I’m going to try what you’re describing, implementing a custom version of the SqlAlchemyExecutionEngine, overriding the default methods. I’ll post here if I get it working.

1 Like