Problem: validator does not work on a shared high concurrency cluster

GX OSS Support
,
1

Hello,

I’m running GX on a shared cluster that has not enabled credential-passthrough for user-level data access. This code snippet:

validator = context.get_validator(
batch_request=batch_request, expectation_suite_name=“test_gcs_suite”
)

produces the following error:

py4j.security.Py4JSecurityException: Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.

Is there any workaround for this?

Nadya

2

Hi @Nadya This isn’t a GX issue but has to do with running Spark on Databricks. I would recommend looking into there documentation to solve this problem.

I’ve some here: pyspark - Error running Spark on Databricks: constructor public XXX is not whitelisted - Stack Overflow