GX with Databricks and Azure Blob Storage

Hello @Chijioke ,

regarding your “late” response - no pressure :slight_smile:

The content of your context object looks pretty much the same as mine, except my
“root_directory”: “/dbfs/mnt/sdl/ … DataQuality/GX/”
is a mount point where your root_directory leads to a Volume because you have UC enabled Databricks workspace (and you can’t use mount points any more afaik!?).

Regarding your context object following question:
Is there a “checkpoints” directory in your “/Volumes/…” folder?
Have you tried the profiling on any dataframe which would create a .yml file in your checkpoints subdir and a .json file in your expectations subdir by executing the last (3rd) code snippet in post GX-Databricks:Datasource-Data asset - Validator - #4 by hdamczy ?

Regarding the multiple directories inside the $web container I’m afraid the issue I created for this which can be found in this post: Data Docs in Azure ADLS $web subdirectory not working - #10 by hdamczy is still not fixed yet.

Greetings, Holger

1 Like