Limited Data Sources in Expectation Gallery: Status of Apache Spark Support?

GX Core Support
1

Hello GreatExpectations Community,

I’ve noticed that the Expectation Gallery now appears to be limited to just two data sources: Postgres and Snowflake. This change has made me curious about the current status of Apache Spark support within GreatExpectations.

Has there been any official update or announcement regarding the support for Apache Spark? Specifically, is support for Spark being phased out, or are there any significant changes affecting Spark users in the latest versions?

This is crucial for our projects as we rely heavily on Spark. Understanding these changes will help us plan our next steps accordingly.

Any insights or pointers to official communications would be greatly appreciated. Thanks in advance!

Additional Context

Current Link to GX Gallery : Great Expectations Gallery of Expectations and Packages
Archived Link to GX Gallery : Great Expectations Gallery of Expectations and Packages

2

Hey @hbshashidhar,

We will continue to support Spark. The new Expectation Gallery will be updated to reflect our support for Spark (and Panda) in the coming few days.

Josh

3

Thanks @joshzheng ! Appreciate your timely response.

1 Like