Using a GX across multiple technical platforms

GX Core Support
1

I checked the docs and the web and can not find a compressive guide or best practice on that.

The reality of data platforms these days is, that things are spread across multiple platforms. Especially cloud and on prem.

From my understanding I would be able to build a common repo of all my suites and this way have a one stop shop of DQ BUT these would now need to run across different platforms. Maybe some will run in Composer, some will run in a custom on prem orchestrator and some might even be run via cron or such.

Is there any built in way to still collect the data docs and results into a common store like GCS although they come asynchronously and spread from 3-4 different platforms and respective runs?

2

Yes, assuming that I understood the question right, this is very much possible.

You can set GX to store the validation results as well as the Data Docs for example in GCS.

Do these documentation pages help?
Validation Store: Configure Validation Result Stores | Great Expectations
Data Docs: Host and share Data Docs | Great Expectations

Do let me know if these help or if you have any questions

1 Like