import great_expectations as ge
print(ge.version)
→ 0.18.19
When running a checkpoint in spark I’m getting some very strange behavior. The checkpoint located my expectations, attempts to run but only makes it rough halfway and exits skipping all spark jobs / stages. No GX error is given… my only clues are a deprecation warning and a MarkSafe message that is vague at best:
2024-07-26 15:53:51,195 - WARNING - /databricks/python/lib/python3.10/site-packages/jinja2/environment.py:1088: DeprecationWarning: 'soft_unicode' has been renamed to 'soft_str'. The old name will be removed in MarkupSafe 2.1.
return concat(self.root_render_func(self.new_context(vars)))
Currently only testing against 215k small records, only 10 expectations and a very simple checkpoint config:
{
"action_list": [
{
"name": "update_data_docs",
"action": {
"class_name": "UpdateDataDocsAction"
}
}
],
"batch_request": {
"datasource_name": "memory_datasource",
"data_asset_name": "complex",
"options": {}
},
"class_name": "Checkpoint",
"config_version": 1.0,
"evaluation_parameters": {},
"expectation_suite_name": "complex",
"module_name": "great_expectations.checkpoint",
"name": "checkpoint_complex",
"profilers": [],
"run_name_template": "%Y%m%d-%H%M%S-my-run-name-template",
"runtime_configuration": {
"result_format": {
"result_format": "COMPLETE",
"unexpected_index_column_names": [
"point_id"
],
"return_unexpected_index_query": true
}
},
"validations": []
}
Any ideas? Any tricks to getting GX to give feedback about its progress? I have logging on but don’t see any other logging messages other than the WARNING.
