GX/Airflow integration demo DAG not quite working

GX Core Support
1

Hi, I’m Andy Webster and I work as a Data Engineer for DeLaRue. We are in the process of building (initially, for POC) an on-premise, FOSS Data Lakehouse using Datahub / Nifi / Airflow / MariaDB / HDFS / Spark / Delta Lake.

12:14

I’m interested in using GX to help build pipelines with the best data quality possible

12:17

I can’t quite get the GX/Airflow integration demo to work, the DAG runs, it builds the strawberry table but the GX task fails to validate due to PermissionError: [Errno 13] Permission denied: ‘/usr/local/airflow/include/great_expectations/uncommitted/validations/strawberry_suite’ ? any ideas why? I have a suspicion the Astro Dev Start which spins up the docker containers has somehow ended up with invalid file permissions on some folders but dont know how to resolve.

12:17

Any help appreciated

2

Sorted! just had to bash into running airflow webserver container and sudo chmod 777 -R /usr/local/airflow

1 Like
3

great! Thank you for sharing