I’m running apache airflow 2.6.1 on an ubuntu VM inside docker. My gx stores are in a MinIO database and airflow and MinIO have access to each other.
When I run the following python code from an outside system running ubuntu I get no issues:
Blockquote
config_data_docs_sites = {
“s3_site”: {
“class_name”: “SiteBuilder”,
“store_backend”: {
“class_name”: “TupleS3StoreBackend”,
“bucket”: “great-expectations”,
“prefix”: “data_docs”,
“boto3_options”: boto3_options
}
},
}
data_context_config = DataContextConfig(store_backend_defaults=S3StoreBackendDefaults(default_bucket_name=GX_BUCKET_NAME),
data_docs_sites=config_data_docs_sites
)
context = BaseDataContext(project_config=data_context_config)
checkpoint = gx.checkpoint.SimpleCheckpoint(
name=f"{asset_name.replace(‘/’, ‘_’)}_test",
data_context=context,
validations=[
{
“batch_request”: my_batch_request,
“expectation_suite_name”: expectation,
},
],
)
print("running expectations on: ", asset_name)
checkpoint.run()
Blockquote
All the validations are saved inside the validation store on MinIO. However running this inside airflow gives me the NoCredentialsError. The logs in airflow show that the datasource and all data assets are being saved. The error occurs during “checkpoint.run()”.