We are currently looking for a data quality framework, and GE is on the list.
Our ETL is written in PySpark. Are we able to embed GE inside our current PySpark scripts, so we can perform ETL and DQ within the same process?
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| Can GE access / validate data from Spark, stored in an S3 bucket? | 1 | 642 | April 20, 2021 | |
| No working examples of Custom expectations on Spark | 0 | 553 | June 29, 2022 | |
| How to install GE in AWS | 0 | 580 | May 11, 2021 | |
| Pros and Cons of implementing DQ within Data Flow vs after the data is loaded in Datalake | 2 | 962 | September 12, 2020 | |
| How to create custom Expectations for Spark | 9 | 2656 | June 3, 2021 |