GX/Airflow integration demo DAG not quite working

Hi, I’m Andy Webster and I work as a Data Engineer for DeLaRue. We are in the process of building (initially, for POC) an on-premise, FOSS Data Lakehouse using Datahub / Nifi / Airflow / MariaDB / HDFS / Spark / Delta Lake.


I’m interested in using GX to help build pipelines with the best data quality possible


I can’t quite get the GX/Airflow integration demo to work, the DAG runs, it builds the strawberry table but the GX task fails to validate due to PermissionError: [Errno 13] Permission denied: ‘/usr/local/airflow/include/great_expectations/uncommitted/validations/strawberry_suite’ ? any ideas why? I have a suspicion the Astro Dev Start which spins up the docker containers has somehow ended up with invalid file permissions on some folders but dont know how to resolve.


Any help appreciated

Sorted! just had to bash into running airflow webserver container and sudo chmod 777 -R /usr/local/airflow

1 Like

great! Thank you for sharing