ERROR: Could not create .gitignore in Read-only file system

GX Core Support
1

Dear team,

We are currently using Great Expectations (GX) version 1.4.6 in our firm. During initialization, a .gitignore file is created in the gx folder. This file is included in our code repository and pushed through the CI/CD pipeline. However, the .gitignore file is excluded from the deployment package, which is expected behavior for CI/CD pipelines.

The issue arises when the deployed code runs in a read-only environment. If GX does not find the .gitignore file, it attempts to create one, resulting in an error: “Could not create .gitignore in Read-only file system.”

Since the deployment environment is intentionally read-only, we need a way to prevent GX from trying to write to the directory. Can this behavior be optimized to avoid such errors?

Below is the path of ge source code when it raised the error.

great_expectations/data_context/data_context/serializable_data_context.py

line 224

2

Hi Jimmy - looks like you’re trying to use a FileDataContext in a read-only environment. That’s likely to cause issues, since the FileDataContext writes config and validation results to disk. Sounds like an EphemeralDataContext might fit your use case better:

import great_expectations as gx
context = gx.get_context(mode="ephemeral")

If your GX project lives in version control, it might be easiest to define your config using the Python API. You can also pass an existing config into the project_config parameter of get_context, if you prefer to define it as a dict.

3

Hi @joshstauffer I have figured out the root cause when I’m using the 0.18.22 version. It may has the issue with the V1. I hope this found will give some suggestion to the whole community and help the people who has the similar issue.

We have the gx project in the firm, then we will release the project into a read-only file system. From there, we will run the project.

We already have the gx folder in our project and it have all the gx configs we need. In the great_expectations.ymal file ,we config the variables_file as this:

config_variables_file_path: ${GX_OUTPUT_FOLDER}/config_variables.yml

When I init a file context. It goes to _scaffold_project method from
great_expectations/great_expectations/data_context/data_context/file_data_context.py

def _scaffold_project(self) -> None:
    """Prepare a `great_expectations` directory with all necessary subdirectories.
    If one already exists, no-op.
    """
    if self.is_project_scaffolded(self._context_root_directory):
        return

    # GX makes an important distinction between project directory and context directory.
    # The former corresponds to the root of the user's project while the latter
    # encapsulates any config (in the form of a great_expectations/ directory).
    project_root_dir = pathlib.Path(self._context_root_directory).parent
    self._scaffold(
        project_root_dir=project_root_dir,
    )

Then is_project_scaffolded method is called from great_expectations/great_expectations/data_context/data_context/serializable_data_context.py

def is_project_initialized(cls, ge_dir: PathStr) -> bool:
    """
    Return True if the project is initialized.

    To be considered initialized, all of the following must be true:
    - the project must be scaffolded (see cls.is_project_scaffolded)
    - the project has at least one datasource
    - the project has at least one suite
    """
    return (
        cls.is_project_scaffolded(ge_dir)
        and cls._does_context_have_at_least_one_datasource(ge_dir)
        and cls._does_context_have_at_least_one_suite(ge_dir)
    )

When is_project_scaffolded is called, config_variables_yml_exist method will be called, then the issue happened. It did not convert the ${GX_OUTPUT_FOLDER} into a real path(we set the value in the python env). So, the config_variables_yml_exist return False, then it will go to create the .gitignore file in the gx folder.

I log the path of config_var_path, it’s showing value of “${GX_OUTPUT_FOLDER}/config_variables.yml” instead of path “real_path/config_variables.yml”

My current fix is replace the ${GX_OUTPUT_FOLDER} with an actual value in the great_expectations.yml file.