I have to validate my data that is present on databricks unity catalog and then have to send the validation result to datahub. I am trying this: context_root_dir = "/dbfs/gx/" context = gx.get_context(context_root_dir=context_root_dir) dataframe_datasource = context.sources.add_or_update_spark( …

Hi @sant-singh , thanks for the question! Would you be able to provide what version of GX you’re using along with the rest of your stack?

Hi @nevintan , Yes, I am using latest version of GX pip install great-expectations using this

It looks like DataHubValidationAction is set up to only use SqlAlchemyExecutionEngine. Have you configured your Data Context accordingly?

Datahub only support SqlAlchemyExecutionEngine https://datahubproject.io/docs/metadata-ingestion/integration_docs/great-expectations/ Note: @Sant Narayan Singh originally posted this reply in Slack . It might not have transferred perfectly.

Can you provide a datasource congiguration with asset using SqlAlchemyExecutionEngine?

Yes, I have seen that and It works well with postgresql but I have my delta tables in Databricks Unity Catalog(not in Databricks SQL warehouse). I am able to fetched that data and generated validation result but I am not able to send that validation result to datahub.

How to integrate great-expectations with databricks and then to datahub

GX Core Support

nevintan November 2, 2023, 4:18pm 7

Have you taken a look at the attached loom video provided in the Datahub documentation?

Topic		Replies	Views
Integrating Great Expectations with Databricks and Datahub GX Core Support how-to , help-wanted , databricks	1	425	October 30, 2023
GX-Databricks:Datasource-Data asset - Validator GX Core Support databricks , datasource	8	586	December 19, 2024
Integration with Datahub: does not work GX Core Support	3	483	September 15, 2023
How to connect to data on Databricks Unity Catalog using Spark GX Core Support how-to , help-wanted , databricks , datasource	3	1433	November 2, 2023
Integration GE with Datahub - execution delete DataHubValidationAction GX Core Support	0	195	August 29, 2023

How to integrate great-expectations with databricks and then to datahub

Related topics