Problem: validator does not work on a shared high concurrency cluster


I’m running GX on a shared cluster that has not enabled credential-passthrough for user-level data access. This code snippet:

validator = context.get_validator(
batch_request=batch_request, expectation_suite_name=“test_gcs_suite”

produces the following error: Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.

Is there any workaround for this?


Hi @Nadya This isn’t a GX issue but has to do with running Spark on Databricks. I would recommend looking into there documentation to solve this problem.

I’ve some here: pyspark - Error running Spark on Databricks: constructor public XXX is not whitelisted - Stack Overflow