# Unable to run Great Expectations on PySpark using SparkDFExecutionEngine

**URL:** <https://discourse.greatexpectations.io/t/unable-to-run-great-expectations-on-pyspark-using-sparkdfexecutionengine/1805>\
**Category:** GX Core Support\
**Created:** [July 12, 2024, 9:06pm UTC](https://discourse.greatexpectations.io/t/unable-to-run-great-expectations-on-pyspark-using-sparkdfexecutionengine/1805 "2024-07-12T21:06:28Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![mshruthi28](https://avatars.discourse-cdn.com/v4/letter/m/3ab097/32.png) [@mshruthi28](https://discourse.greatexpectations.io/u/mshruthi28)\
**Post date:** [July 12, 2024, 9:06pm UTC](https://discourse.greatexpectations.io/t/unable-to-run-great-expectations-on-pyspark-using-sparkdfexecutionengine/1805/1 "2024-07-12T21:06:28Z")

</div>

Hi Team,

I am exploring Great Expectations to use it as a Data Quality tool for our datalake which is in delta format in GCS bucket. I am planning to use Composer for Orchestration and use a DataprocSubmitPySparkJobOperator to trigger the python job.

I have a simple python job like below  
def run\_great\_expectations(context\_root\_dir, checkpoint\_name):  
context = DataContext(context\_root\_dir=context\_root\_dir)  
print(context.datasources.values())  
checkpoint\_result = context.run\_checkpoint(checkpoint\_name=checkpoint\_name)  
print(checkpoint\_result)

Below is my great\_expectations.yml for the datasoource: spark\_delta\_read

```auto
datasources:
  spark_delta_read:
    class_name: Datasource
    module_name: great_expectations.datasource
    execution_engine:
      class_name: SparkDFExecutionEngine
      module_name: great_expectations.execution_engine
    data_connectors:
      default_inferred_data_connector_name:
        name: default_inferred_data_connector_name
        class_name: InferredAssetFilesystemDataConnector
        module_name: great_expectations.datasource.data_connector
        base_directory: gs://bucket/data/
        default_regex:
          group_names:
            - data_asset_name
          pattern: (.*)
        batch_spec_passthrough:
          reader_method: delta

```

My checkpoint file looks like below

```auto
name: brands_delta_checkpoint
config_version: 1.0
class_name: SimpleCheckpoint
validations:
  - batch_request:
      datasource_name: spark_delta_read
      data_connector_name: default_inferred_data_connector_name
      data_asset_name: brands
      data_connector_query:
        index: -1
action_list:
  - name: store_validation_result
    action:
      class_name: StoreValidationResultAction
  - name: store_evaluation_params
    action:
      class_name: StoreEvaluationParametersAction
  - name: update_data_docs
    action:
      class_name: UpdateDataDocsAction        
expectation_suite_name: brands

```

As soon I load my context, I get the error great\_expectations.exceptions.exceptions.DatasourceInitializationError: Cannot initialize datasource spark\_delta\_read, error: ‘spark.remote’

Below are the versions I am using  
great\_expectations : 0.18.7  
pyspark : 3.4.2

Please guide me with the issue and the suggest me if I have to modify anything in my great\_expectations.yml file