Hi,
We are upgrading Great Expectations to 0.18.10. We are currently experiencing a blocker issue from one of the packages that throws a read-only file system error.
great-expectationsversion 0.18.10- airflow-provider-great-expectations 0.2.8
- Airflow: Version: v2.8.1
Can someone please help how can we resolve this error?
config file:
datasources:
sandbox:
data_connectors:
default_runtime_data_connector_name:
batch_identifiers:
- default_identifier_name
name: default_runtime_data_connector_name
class_name: RuntimeDataConnector
module_name: great_expectations.datasource.data_connector
default_inferred_data_connector_name:
include_schema_name: true
name: default_inferred_data_connector_name
class_name: InferredAssetSqlDataConnector
module_name: great_expectations.datasource.data_connector
execution_engine:
credentials:
host: ${host}
port: 4343
username: secret
password: secret
database: db
query:
sslmode: prefer
drivername: postgresql+psycopg2
module_name: great_expectations.execution_engine
class_name: SqlAlchemyExecutionEngine
module_name: great_expectations.datasource
class_name: Datasource
config_variables_file_path: ${AIRFLOW_HOME}/dags/data-quality/config_variables/sandbox/config_variables.yml
stores:
expectations_store:
class_name: ExpectationsStore
store_backend:
class_name: TupleFilesystemStoreBackend
base_directory: expectations/
validations_filesystem_store:
class_name: ValidationsStore
store_backend:
class_name: TupleS3StoreBackend
bucket: data-sandbox-great-expectations
prefix: data-quality/validations/
evaluation_parameter_store:
class_name: EvaluationParameterStore
checkpoint_store:
class_name: CheckpointStore
store_backend:
class_name: TupleFilesystemStoreBackend
suppress_store_backend_id: true
base_directory: checkpoints/
profiler_store:
class_name: ProfilerStore
store_backend:
class_name: TupleFilesystemStoreBackend
suppress_store_backend_id: true
base_directory: /tmp/great_expectations/profilers/
expectations_store_name: expectations_store
validations_store_name: validations_filesystem_store
evaluation_parameter_store_name: evaluation_parameter_store
checkpoint_store_name: checkpoint_store
data_docs_sites:
local_site:
class_name: SiteBuilder
show_how_to_buttons: true
store_backend:
class_name: TupleS3StoreBackend
bucket: data-sandbox-great-expectations
prefix: data-quality/data_docs/local_site/
site_index_builder:
class_name: DefaultSiteIndexBuilder
anonymous_usage_statistics:
data_context_id: ddf20563-8d61-4ebe-9814-2hg5115c4710
enabled: false
notebooks:
include_rendered_content:
globally: false
expectation_suite: false
expectation_validation_result: false
progress_bars:
globally: false
profilers: false
metric_calculations: false
Error Logs:
[2024-04-12, 03:09:46 UTC] {{taskinstance.py:2480}} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='data-model-quality' AIRFLOW_CTX_TASK_ID='accounts' AIRFLOW_CTX_EXECUTION_DATE='2024-04-12T03:08:55.420641+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2024-04-12T03:08:55.420641+00:00'
[2024-04-12, 03:09:46 UTC] {{great_expectations.py:580}} INFO - Running validation with Great Expectations...
[2024-04-12, 03:09:46 UTC] {{great_expectations.py:582}} INFO - Instantiating Data Context...
[2024-04-12, 03:09:46 UTC] {{taskinstance.py:2698}} ERROR - Task failed with exception
Traceback (most recent call last):
File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 433, in _execute_task
result = execute_callable(context=context, **execute_callable_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/airflow/.local/lib/python3.11/site-packages/great_expectations_provider/operators/great_expectations.py", line 586, in execute
self.data_context = ge.data_context.FileDataContext(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/airflow/.local/lib/python3.11/site-packages/great_expectations/data_context/data_context/file_data_context.py", line 64, in __init__
self._scaffold_project()
File "/usr/local/airflow/.local/lib/python3.11/site-packages/great_expectations/data_context/data_context/file_data_context.py", line 98, in _scaffold_project
self._scaffold(
File "/usr/local/airflow/.local/lib/python3.11/site-packages/great_expectations/data_context/data_context/serializable_data_context.py", line 219, in _scaffold
gx_dir.mkdir(parents=True, exist_ok=True)
File "/usr/local/lib/python3.11/pathlib.py", line 1116, in mkdir
os.mkdir(self, mode)
OSError: [Errno 30] Read-only file system: '/usr/local/airflow/dags/data-model-schema-data-quality/gx'
[2024-04-12, 03:09:46 UTC] {{taskinstance.py:1138}} INFO - Marking task as FAILED. dag_id=data-model-quality, task_id=accounts, execution_date=20240412T030855, start_date=20240412T030946, end_date=20240412T030946
[2024-04-12, 03:09:47 UTC] {{standard_task_runner.py:107}} ERROR - Failed to execute job 37515 for task accounts ([Errno 30] Read-only file system: '/usr/local/airflow/dags/data-quality/gx'; 3080)
[2024-04-12, 03:09:47 UTC] {{local_task_job_runner.py:234}} INFO - Task exited with return code 1
[2024-04-12, 03:09:47 UTC] {{taskinstance.py:3280}} INFO - 0 downstream tasks scheduled from follow-on schedule check
