Hi All,
I have observed that there are no working examples of custom expectations (Created locally and not submitted) for Spark.
For Pandas it is easy you can do ge.from_pandas and can give the custom class there. SAme cannot be done for Spark.
There is thid doc : How to create custom Expectations for Spark
However as soon as you add the following to the datasource config
data_asset_type:
module_name: my_custom_expectations
class_name: CustomSparkDFDataset
it fails, it does not recognize data asset type.
Could someone please help urgently.