[Updated] File not found error : Unable to generate result json files and data docs

[Updated with new test on notebook] When I try to run the following code I expect the results to be generated inside the json files and stored inside the validations folder as well the data docs stored inside the data docs folder. I am using the same block of code on Pycharm as python script and on jupyter notebook.

Jupyter notebook:The same code in notebook returns the data docs files as well as the json files for the previously configured expectation suite test_example_3 but when I try to configure a new expectation suite WH29_tyrewear_expectation_set and run the validation over them in notebook data docs are not generated but json files are generated.

Pycharm : The same code in pycharm returns the json files for the previously configured expectation suite test_example_3 but when I try to configure a new expectation suite WH29_tyrewear_expectation_set and validation over them no json files and no data docs are generated.

Code :

import great_expectations as ge
from great_expectations.core.expectation_configuration import (ExpectationConfiguration)
context = ge.data_context.DataContext()

datasource_name = “WH29_measurement_dates”
path_to_folder_containing_csv_files = “Combined_singleday_data_for_each_measurement_dates”

datasource = context.sources.add_or_update_pandas_filesystem(name=datasource_name, base_directory=path_to_folder_containing_csv_files)
#Set the asset name as well as the regular expression for the dates when data is available

asset_name = “measurement_data_eachday”
batching_regex = r"37367b5d-daa9-4a6b-a6d7-683cddca44e0_(?P\d{4})-(?P\d{2})-(?P\d{2}).csv"
datasource.add_csv_asset(name=asset_name, batching_regex=batching_regex)
my_asset = context.get_datasource(“WH29_tyrewear_measurement_dates”).get_asset(“WH29_tyrewear_measurement_data_eachday”)
context.add_expectation_suite(expectation_suite_name=“WH29_tyrewear_expectation_set”)
context.get_expectation_suite(“WH29_tyrewear_expectation_set”)
batch_request = my_asset.build_batch_request()

validator = context.get_validator(batch_request=batch_request,expectation_suite_name=“WH29_tyrewear_expectation_set”)
expectation_validation_temp_r2 =validator.expect_column_values_to_be_between(“tire_pressure_controller_tire_temperature_r2”,min_value=0,max_value = 70,strict_min = True,strict_max=True)
validator.save_expectation_suite(discard_failed_expectations=False)
print(expectation_validation_temp_r1)
checkpoint = context.add_or_update_checkpoint(
name=“WH29_tyrewear_expectation_set”,
run_name_template =“WH29_tyrewear_trial_2”,
validations=[
{
“batch_request”: batch_request,
“expectation_suite_name”: “WH29_tyrewear_expectation_set”,
},
],
)
checkpoint_result = checkpoint.run()

In Image 1 it can be noticed that the folders for the newly created expectation suite and checkpoint is empty and the folders with previously made expectations suite are with json files whereas the data_docs is empty for everything.

Expected behaviour after running checkpoint:

  1. Build json files
  2. Build data_docs

The result I got for running the new expectation suite on Pycharm:

Error running action with name store_validation_result
Traceback (most recent call last):
File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\validation_operators\validation_operators.py”, line 478, in _run_actions
 action_result = self.actions[name].run(
File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\checkpoint\actions.py”, line 100, in run
return self._run(
File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\checkpoint\actions.py”, line 885, in _run
run_return_value = self._basic_run(
 File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\checkpoint\actions.py”, line 931, in _basic_run
return self.target_store.set(
File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\data_context\store\store.py”, line 226, in set
return self._store_backend.set(
File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\data_context\store_store_backend.py”, line 134, in set
return self._set(key, value, **kwargs)
File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\data_context\store\tuple_store_backend.py”, line 342, in _set
with open(filepath, “wb”) as outfile:
FileNotFoundError: [Errno 2] No such file or directory: ‘C:\Users\memonju\PycharmProjects\great_expectations_eg\gx\uncommitted/validations/WH29_tyrewear_expectation_set\WH29_tyrewear_trial_2\20240312T155739.587972Z\WH29_tyrewear_measurement_dates_2-WH29_tyrewear_measurement_data_eachday_2-year_2024-month_01-day_20.json’
Traceback (most recent call last):
 File “C:\Users\memonju\PycharmProjects\great_expectations_eg\great_expectations_eg.py”, line 102, in
checkpoint_result = checkpoint.run()
File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\core\usage_statistics\usage_statistics.py”, line 266, in usage_statistics_wrapped_method
result = func(<em>args, **kwargs)
File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\checkpoint\checkpoint.py”, line 305, in run</em>*
self._run_validation(
File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\checkpoint\checkpoint.py”, line 530, in _run_validation**
 async_validation_operator_result = async_executor.submit(
File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\core\async_executor.py”, line 106, in submit
return AsyncResult(value=fn(<em>args, **kwargs))
File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\validation_operators\validation_operators.py”, line 415, in run</em>*
batch_actions_results = self._run_actions(
File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\validation_operators\validation_operators.py”, line 508, in _run_actions**
raise e
 File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\validation_operators\validation_operators.py”, line 478, in _run_actions
action_result = self.actions[name].run(
 File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\checkpoint\actions.py”, line 100, in run**
 return self._run(
File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\checkpoint\actions.py”, line 885, in _run
run_return_value = self._basic_run(
File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\checkpoint\actions.py”, line 931, in _basic_run
return self.target_store.set(
File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\data_context\store\store.py”, line 226, in set
 return self._store_backend.set(
File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\data_context\store_store_backend.py”, line 134, in set
return self._set(key, value, **kwargs)
File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\data_context\store\tuple_store_backend.py”, line 342, in _set
 with open(filepath, “wb”) as outfile:
FileNotFoundError: [Errno 2] No such file or directory: ‘C:\Users\memonju\PycharmProjects\great_expectations_eg\gx\uncommitted/validations/WH29_tyrewear_expectation_set\WH29_tyrewear_trial_2\20240312T155739.587972Z\WH29_tyrewear_measurement_dates_2-WH29_tyrewear_measurement_data_eachday_2-year_2024-month_01-day_20.json’

Environment :
OS : Windows 11
Great Expectations Version : 0.18.10
Python Version : 3.9.0
IDE : Pycharm & Jupyter Notebook

Error I receive for the previously configured expectation suite from which json files are generated and not the data docs on both IDE’s :

An unexpected Exception occurred during data docs rendering. Because of this error, certain parts of data docs will not be rendered properly and/or may not appear altogether. Please use the trace, included in this message, to diagnose and repair the underlying issue. Detailed information follows:
FileNotFoundError: “[Errno 2] No such file or directory: ‘C:\Users\memonju\PycharmProjects\great_expectations_eg\gx\uncommitted/data_docs/local_site/validations\test_example_3\WH29_tyrewear_trial_3\20240312T160338.728982Z\WH29_tyrewear_measurement_dates_2-WH29_tyrewear_measurement_data_eachday_2-year_2024-month_01-day_20.html’”. Traceback: "Traceback (most recent call last):
File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\render\renderer\site_builder.py”, line 493, in build
self.target_store.set(
File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\data_context\store\html_site_store.py”, line 269, in set
return self.store_backends[type(key.resource_identifier)].set(
File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\data_context\store_store_backend.py”, line 134, in set
return self._set(key, value, **kwargs)
File “C:\Users\memonju\PycharmProjects\Lubricant Control Web\venv\lib\site-packages\great_expectations\data_context\store\tuple_store_backend.py”, line 342, in _set
with open(filepath, “wb”) as outfile:
FileNotFoundError: [Errno 2] No such file or directory: ‘C:\Users\memonju\PycharmProjects\great_expectations_eg\gx\uncommitted/data_docs/local_site/validations\test_example_3\WH29_tyrewear_trial_3\20240312T160338.728982Z\WH29_tyrewear_measurement_dates_2-WH29_tyrewear_measurement_data_eachday_2-year_2024-month_01-day_20.html’**
".


Process finished with exit code 0

Is there possibility to solve this error or is it some bug ?

Image 1 :

This might be as simple as the file paths being too long for Windows.

Windows has a limit on the path length of roughly 256 characters which can cause some weird issues sometimes.

I.e. I couldn’t open the Data Docs without moving them to Desktop first.

Note that this is only a guess. You mentioned that the Data Docs and validation JSONs are generated just fine when the file paths are shorter, which would match my assumption.

Couple of ways you can try to figure out if this is the problem:

  • Change the Expectation Suite etc names to be shorter
  • Move the project folder so that the path is shorter, i.e. to C:/ (or D:/ if you have another drive available)
  • Run the code on another operating system (MacOS, Linux, WSL, etc) if you have those available

Also, please note that GX is not officially supported on Windows
It works but you can run against weird problems, like the path length limit.
I run GX in Windows Subsystem for Linux (WSL) and it works fine for me, haven’t had these kinds of problems.