# GX with Databricks and Azure Blob Storage

**URL:** <https://discourse.greatexpectations.io/t/gx-with-databricks-and-azure-blob-storage/1777>\
**Category:** GX Core Support\
**Created:** [June 18, 2024, 9:03am UTC](https://discourse.greatexpectations.io/t/gx-with-databricks-and-azure-blob-storage/1777 "2024-06-18T09:03:23Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Chijioke](https://avatars.discourse-cdn.com/v4/letter/c/d07c76/32.png) [@Chijioke](https://discourse.greatexpectations.io/u/Chijioke)\
**Post date:** [June 18, 2024, 9:03am UTC](https://discourse.greatexpectations.io/t/gx-with-databricks-and-azure-blob-storage/1777/1 "2024-06-18T09:03:23Z")

</div>

Hello All,

I have been working with the GX package for few days now but I have a series of question to ask:  
I want to ask whether it is possible to do this:

1. Initiate a file system data context directly in a container in Azure blob storage instead of using DBFS root in databricks. If this is possible, can someone guide me on how to implement this?
2. If the Above is not possible, when we want to host and share datadocs in Azure blob storage, MUST I configure other artifacts such as expectation store, checkpoint stores etc, to be in the Azure blob storage too OR can I leave them in dbfs root and ONLY configure the hosting and sharing of docs in the blob storage?

---

<div class="post-metadata">

**Author:** ![hdamczy](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/hdamczy/32/245_2.png) [@hdamczy](https://discourse.greatexpectations.io/u/hdamczy)\
**Post date:** [June 18, 2024, 9:29am UTC](https://discourse.greatexpectations.io/t/gx-with-databricks-and-azure-blob-storage/1777/2 "2024-06-18T09:29:22Z")

</div>

Hi @Chijioke ,

have you seen my post here:

> [@GX-Databricks:Datasource-Data asset - Validator](https://discourse.greatexpectations.io/t/gx-databricks-datasource-data-asset-validator/1738/4):
>
> Hi @kat , I use Databricks in conjunction with ADLS. Here’s the Code I use for this: Class in our functions .whl file: import great\_expectations as gx class GX\_Context: def \_\_init\_\_(self, root\_dir, connection\_string, filepath\_prefix=""): self.root\_dir = root\_dir self.connection\_string = connection\_string self.filepath\_prefix = filepath\_prefix if not isinstance(self.root\_dir, str) or not self.root\_dir: raise ValueError("root\_dir must be a non…

This should answer your 1st question I guess. 🙂

If you have more detailed questions, please ask.

---

<div class="post-metadata">

**Author:** ![Chijioke](https://avatars.discourse-cdn.com/v4/letter/c/d07c76/32.png) [@Chijioke](https://discourse.greatexpectations.io/u/Chijioke)\
**Post date:** [June 18, 2024, 11:39am UTC](https://discourse.greatexpectations.io/t/gx-with-databricks-and-azure-blob-storage/1777/3 "2024-06-18T11:39:28Z")

</div>

Hello @hdamczy , Thank you for your response.

Correct me if I am misreading the code.  
Your context\_root\_dir is still pointing to dbfs root and the datadocs is the only part that is actually stored and hosted on azure Storage right?

---

<div class="post-metadata">

**Author:** ![hdamczy](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/hdamczy/32/245_2.png) [@hdamczy](https://discourse.greatexpectations.io/u/hdamczy)\
**Post date:** [June 18, 2024, 12:08pm UTC](https://discourse.greatexpectations.io/t/gx-with-databricks-and-azure-blob-storage/1777/4 "2024-06-18T12:08:18Z")

</div>

I’m not sure to answer your question 100% precisely but afaik due to I create a EphemeralDataContext no gx yml file is stored anywhere at the beginning.  
context\_root\_dir is a mount point to an ADLS container which is looks like this: /dbfs/mnt/myproject/DataQuality/GX/  
Afaik you can’t have a direct link to your ADLS (maybe it’s possible these days?) so we solved this problem using mountpoints. All data is stored in ADLS containers. The expectations suites .json files and the checkpoint .yml files as well as all data docs files in the $web container.

---

<div class="post-metadata">

**Author:** ![Chijioke](https://avatars.discourse-cdn.com/v4/letter/c/d07c76/32.png) [@Chijioke](https://discourse.greatexpectations.io/u/Chijioke)\
**Post date:** [June 19, 2024, 7:22am UTC](https://discourse.greatexpectations.io/t/gx-with-databricks-and-azure-blob-storage/1777/5 "2024-06-19T07:22:08Z")

</div>

Hello @hdamczy

I don’t think we can have a direct link to ADLS, I used a mounted volume equally. I noticed something after I configured the data context.

1. The config\_variables.yml and great\_expectations.yml files are both missing in the ADLS container where other gx contents are seen. Is this a problem?

2. I configured the context to equally keep the contents of datadocs inside the $web container which it actually does, but when I copy the primary endpoint url  
and try to execute it on a browser, the page does not load. It displays an authentication error message.

Can you check whether I am missing something?  
Below is the config:

```auto
root_dir="/Volumes/test/dq/"

data_context_config = gx.data_context.types.base.DataContextConfig(
        ## Local storage backend
        store_backend_defaults=gx.data_context.types.base.FilesystemStoreBackendDefaults(
            root_directory=root_dir
        ),

        ## Data docs site storage
        data_docs_sites={
            "az_site": {
            "class_name": "SiteBuilder",
            "store_backend": {
                "class_name": "TupleAzureBlobStoreBackend",
                "container": r"\$web",
                "connection_string": "DefaultEndpointsProtocol=https;AccountName=<account_name>;AccountKey=<account_key>;EndpointSuffix=core.windows.net",
            },
            "site_index_builder": {
                "class_name": "DefaultSiteIndexBuilder",
                "show_cta_footer": True,
            },
            }
        },
     )

context = gx.get_context(project_config=data_context_config)

```

---

<div class="post-metadata">

**Author:** ![hdamczy](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/hdamczy/32/245_2.png) [@hdamczy](https://discourse.greatexpectations.io/u/hdamczy)\
**Post date:** [June 19, 2024, 4:16pm UTC](https://discourse.greatexpectations.io/t/gx-with-databricks-and-azure-blob-storage/1777/6 "2024-06-19T16:16:35Z")

</div>

Hello @Chijioke ,

I do not have config\_variables.yml and great\_expectations.yml either.  
Everything GX needs on runtime is created by the function get\_gx\_context() in class GX\_Context. You can put the code from the class directly in your Databricks notebook for testing purposes.

I just had my mount to DataQuality/GX/ on ADLS where 4 directories were created (checkpoints/, expectations/, profilers/ and uncommitted/)

After profiling my dataframes (see code below “To get the expectation-suite JSON files and the checkpoint YML files you can do a profiling…”) I had a .yml in checkpoints/ dir for each dataframe I profiled as well as .json in expectations/ dir.

Both files are loaded during runtime.  
Maybe this schema helps a bit:

 ![image](https://canada1.discourse-cdn.com/flex031/uploads/greatexpectations/original/1X/3174866e301b6dad58480321abcbb303f5c5863e.png)

On your storage account you have to activate the static website like this:

 ![image](https://canada1.discourse-cdn.com/flex031/uploads/greatexpectations/original/1X/d4fa89817c7f21c0fe9e3d7a55b76a69cc5ead2f.png)

Then you can access the website via URL “Primary endpoint” (see pic (4)).

Your “roor\_dir” seems to be a Volume (is UnityCatalog enabled?).  
I do not have Unity enabled and my “root\_dir” (= context\_root\_dir) is a mountpoint to ADLS

Can you sent what’s inside the context-Variable at the end?  
print(context)

---

<div class="post-metadata">

**Author:** ![Chijioke](https://avatars.discourse-cdn.com/v4/letter/c/d07c76/32.png) [@Chijioke](https://discourse.greatexpectations.io/u/Chijioke)\
**Post date:** [June 24, 2024, 2:37pm UTC](https://discourse.greatexpectations.io/t/gx-with-databricks-and-azure-blob-storage/1777/7 "2024-06-24T14:37:37Z")

</div>

Hello @hdamczy,  
Apologies for my late response. Thank you so much for your well detailed last response.  
Below is the image of the context.

 ![image](https://canada1.discourse-cdn.com/flex031/uploads/greatexpectations/original/1X/faeb8e2a4515be2eac5a688df9f6bf36c275e6c8.png)

I got to find out that the issue with the endpoint url was related to a firewall in our network and not with the way it was set up.

I actually have a use case where I need to create multiple directories in the $web container and I think you complained of an issue where you created a sub directory inside the $web container and was unable to view the index.htl file properly. Were you later able to resolve this issue?

---

<div class="post-metadata">

**Author:** ![hdamczy](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/hdamczy/32/245_2.png) [@hdamczy](https://discourse.greatexpectations.io/u/hdamczy)\
**Post date:** [June 24, 2024, 7:22pm UTC](https://discourse.greatexpectations.io/t/gx-with-databricks-and-azure-blob-storage/1777/8 "2024-06-24T19:22:45Z")

</div>

Hello @Chijioke ,

regarding your “late” response - no pressure 🙂

The content of your context object looks pretty much the same as mine, except my  
“root\_directory”: “/dbfs/mnt/sdl/ … DataQuality/GX/”  
is a mount point where your root\_directory leads to a Volume because you have UC enabled Databricks workspace (and you can’t use mount points any more afaik!?).

Regarding your context object following question:  
Is there a “checkpoints” directory in your “/Volumes/…” folder?  
Have you tried the profiling on any dataframe which would create a .yml file in your checkpoints subdir and a .json file in your expectations subdir by executing the last (3rd) code snippet in post [GX-Databricks:Datasource-Data asset - Validator - #4 by hdamczy](https://discourse.greatexpectations.io/t/gx-databricks-datasource-data-asset-validator/1738/4) ?

Regarding the multiple directories inside the $web container I’m afraid the issue I created for this which can be found in this post: [Data Docs in Azure ADLS $web subdirectory not working - #10 by hdamczy](https://discourse.greatexpectations.io/t/data-docs-in-azure-adls-web-subdirectory-not-working/1627/10) is still not fixed yet.

Greetings, Holger

---

<div class="post-metadata">

**Author:** ![Chijioke](https://avatars.discourse-cdn.com/v4/letter/c/d07c76/32.png) [@Chijioke](https://discourse.greatexpectations.io/u/Chijioke)\
**Post date:** [July 2, 2024, 8:02am UTC](https://discourse.greatexpectations.io/t/gx-with-databricks-and-azure-blob-storage/1777/9 "2024-07-02T08:02:44Z")

</div>

Hello @hdamczy  
Thank you 🙂  
Yes, there is a checkpoint and expectations directory created in the container on azure blob storage. I created the my expectations for each data asset interactively with python, all in a databricks notebook. I have not tried profiling yet…

Regarding the multiple directories inside the $web container, I will take a look at it some more and If I find any thing tangible, I will let you know