Problem: GX with databricks dbfs-data Context

GX Core Support
, ,
1

I am trying to create context in the dbfs using the below code:

import great_expectations as gx
from great_expectations import get_context

context = get_context(context_root_dir='/dbfs/great_expectations/")

considering great_expectations/ emplty folder exists in dbfs. Context object is not getting created in the directory.
And when I am trying to create a file data context at the same dbfs location. It is getting created in the drivers memory of databricks cluster.

Could you please put light on this issue.
Thanks

2

Hi @DE_Saheed, thanks for reaching out! Welcome to the GX community.

We had a recent related thread that explored how to get the GX context to write to a DBFS-mounted folder, I think this will have the examples you need to help get your context working on Databricks.

  • Post 4 has my suggested solution and
  • Post 5 has the user’s final solution.

Could you take a look and see if that helps solve your issue?