This section of the migration guide: https://docs.greatexpectations.io/docs/reference/learn/migration_guide#store-backends mentions that GX Core 1.0 drops support for anything other than filesystem store backends.
This is very inconvenient for anyone who wants to easily (without the use of other tools) share a store between multiple cloud and on-premises systems. Is there any plan to change this and re-enable support for the AWS S3 + other store backends?
Hey @scervienka, we didn’t remove any real functionality here besides the convenience methods + documentation. All your config is stored in the config file anyways, and like the guide said, you can use something like boto3/other libraries to store/retrieve the file on S3/other cloud platforms.
Hi Josh, thanks for the reply.
Maybe I’m misunderstanding the migration guide, but I am referring to the store backends for the expectation/validation/checkpoint stores and data docs, not the config yml file itself.
Working with the constraint of a filesystem is easy enough for the config since it’s just one file, but for the expectation/validation/checkpoint stores and data docs it can be difficult to build a shared data quality system across multiple cloud services and premises with the filesystem constraint. It would be much easier if we could just save everything to S3 as we could before.
So are we supposed to still be able to use the TupleS3StoreBackend in 1.0 as a store backend or no?
hi @scervienka, with a FileDataContext, you can copy the files to S3 (or any other storage) if that’s your preferred hosting option. Moving forward, however, we won’t provide cloud-specific backend support. Instead, we’ll focus on developing user guides that show how to use GX Core with popular configuration management and storage tools. Within the product, files will simply be written to disk using the standard FileSystem backend provider. Please note, though, that you’re still free to store them in any location you choose.
Thanks for confirming. It adds some complexity on my end but at least I can build a solution knowing that’s the direction GX is going.