# How to connect to data on Databricks Unity Catalog using Spark

**URL:** <https://discourse.greatexpectations.io/t/how-to-connect-to-data-on-databricks-unity-catalog-using-spark/1463>\
**Category:** GX Core Support\
**Tags:** how-to, help-wanted, databricks, datasource\
**Created:** [October 31, 2023, 11:18am UTC](https://discourse.greatexpectations.io/t/how-to-connect-to-data-on-databricks-unity-catalog-using-spark/1463 "2023-10-31T11:18:23Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sant-singh](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@sant-singh](https://discourse.greatexpectations.io/u/sant-singh)\
**Post date:** [October 31, 2023, 11:18am UTC](https://discourse.greatexpectations.io/t/how-to-connect-to-data-on-databricks-unity-catalog-using-spark/1463/1 "2023-10-31T11:18:23Z")

</div>

I have my data in Databricks Unity Catalog as

```auto
main.checkout_catalog.checkout_schema.checkout_orders_data

```

How can I connect with data present in above given path

---

<div class="post-metadata">

**Author:** ![rachel.house](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/rachel.house/32/236_2.png) [@rachel.house](https://discourse.greatexpectations.io/u/rachel.house)\
**Post date:** [November 1, 2023, 7:31pm UTC](https://discourse.greatexpectations.io/t/how-to-connect-to-data-on-databricks-unity-catalog-using-spark/1463/2 "2023-11-01T19:31:53Z")

</div>

Hi @sant-singh, thanks for reaching out!

If you’d like to use Spark to connect to a Databricks table, you can use `spark.read.table("main.checkout_catalog.checkout_schema.checkout_orders_data")` to read your table into a Spark dataframe. From there, you can use GX to add a Spark data source and a dataframe asset using the newly created dataframe from your table.

Here’s a link to our relevant docs page on connecting to in-memory Data Assets: [Connect to in-memory Data Assets | Great Expectations](https://docs.greatexpectations.io/docs/guides/connecting_to_your_data/fluent/in_memory/connect_in_memory_data), along with some sample code adapted for the `spark.read.table()` usage:

```auto
data_source = context.sources.add_or_update_spark(name="spark_datasource")

df = spark.read.table("main.checkout_catalog.checkout_schema.checkout_orders_data")

data_asset = data_source.add_dataframe_asset(
    name="spark_dataframe_asset",
    dataframe=df
)

batch_request = data_asset.build_batch_request()

```

---

<div class="post-metadata">

**Author:** ![sant-singh](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@sant-singh](https://discourse.greatexpectations.io/u/sant-singh)\
**Post date:** [November 2, 2023, 5:31am UTC](https://discourse.greatexpectations.io/t/how-to-connect-to-data-on-databricks-unity-catalog-using-spark/1463/3 "2023-11-02T05:31:42Z")

</div>

I have delta tables only. I have to use the tables directly not by reading it in dataframe because I have to send the validation result to Datahub and that should match data asset type to be ingested in datahub.

I am getting this while executing the provided sample code:

```auto
DataHubValidationAction does not recognize this GE data asset type - <class 'great_expectations.validator.validator.Validator'>. This is either using v2-api or execution engine other than sqlalchemy.
Metadata not sent to datahub. No datasets found.

```

I have to use only SqlAlchemyExecutionEngine because Datahub only support SqlAlchemyExecutionEngine while integrating Great Expectation and Datahub.

---

<div class="post-metadata">

**Author:** ![rachel.house](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/rachel.house/32/236_2.png) [@rachel.house](https://discourse.greatexpectations.io/u/rachel.house)\
**Post date:** [November 2, 2023, 5:36pm UTC](https://discourse.greatexpectations.io/t/how-to-connect-to-data-on-databricks-unity-catalog-using-spark/1463/4 "2023-11-02T17:36:48Z")

</div>

Hi @sant-singh, if you’re not using Spark, an alternative way to connect to a Databricks table is by [creating a Databricks SQL source and table asset](https://docs.greatexpectations.io/docs/guides/connecting_to_your_data/fluent/database/connect_sql_source_data).

With regard to your Datahub error, I see you’ve also [raised that error in another post](https://discourse.greatexpectations.io/t/how-to-integrate-great-expectations-with-databricks-and-then-to-datahub/1461) - let’s keep conversation around the Datahub error in its original thread. This will help prevent duplicate effort and help keep all the communication & responses consolidated in one place. Thanks!