Dear Support Team,
I am trying to use the ‘expect_column_values_to_be_of_type’ expectation in pyspark for a column of DecimalType. I am facing issues while using the DecimalType with precision and scale parameters.
I am getting following error while using the type as DecimalType(20,0)
{
"success": False,
"expectation_config": {
"expectation_type": "expect_column_values_to_be_of_type",
"meta": {},
"kwargs": {
"column": "project_id",
"type_": "DecimalType(20,0)",
"result_format": {
"result_format": "SUMMARY"
}
}
},
"meta": {},
"exception_info": {
"raised_exception": True,
"exception_message": "ValueError: Unrecognized spark type: DecimalType(20,0)",
"exception_traceback": "Traceback (most recent call last):\n File "/home/spark/.local/lib/python3.7/site-packages/great_expectations/dataset/sparkdf_dataset.py", line 1196, in expect_column_values_to_be_of_type\n success = issubclass(col_type, getattr(sparktypes, type_))\nAttributeError: module \"pyspark.sql.types\" has no attribute \"DecimalType(20,0)\"\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/home/spark/.local/lib/python3.7/site-packages/great_expectations/data_asset/data_asset.py", line 275, in wrapper\n return_obj = func(self, **evaluation_args)\n File "/home/spark/.local/lib/python3.7/site-packages/great_expectations/dataset/sparkdf_dataset.py", line 1201, in expect_column_values_to_be_of_type\n raise ValueError(f"Unrecognized spark type: {type_
}")\nValueError: Unrecognized spark type: DecimalType(20,0)\n"
},
"result": {}
}
There is no error while I use the DecimalType as the value (see below)
{
"exception_info": {
"exception_message": null,
"exception_traceback": null,
"raised_exception": false
},
"expectation_config": {
"expectation_type": "expect_column_values_to_be_of_type",
"kwargs": {
"column": "project_id",
"result_format": {
"result_format": "SUMMARY"
},
"type_": "DecimalType"
},
"meta": {}
},
"meta": {},
"result": {
"observed_value": "DecimalType"
},
"success": true
}
I tried searching documentation but could not find similar topic addressing it.
Does great_expectation support the DecimalType with precision and scale parameter?
Could you please provide your feedback on this topic?
Thanks in advance for your support.