# Help understanding datasource\_name config for RuntimeBatchRequest

**URL:** <https://discourse.greatexpectations.io/t/help-understanding-datasource-name-config-for-runtimebatchrequest/1374>\
**Category:** GX Core Support\
**Tags:** s3, datasource\
**Created:** [September 6, 2023, 7:54am UTC](https://discourse.greatexpectations.io/t/help-understanding-datasource-name-config-for-runtimebatchrequest/1374 "2023-09-06T07:54:16Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![anais](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@anais](https://discourse.greatexpectations.io/u/anais)\
**Post date:** [September 6, 2023, 7:54am UTC](https://discourse.greatexpectations.io/t/help-understanding-datasource-name-config-for-runtimebatchrequest/1374/1 "2023-09-06T07:54:17Z")

</div>

Hi, I am trying to use great expectations with aws glue. The data will be read at runtime, I follow this tutorial [How to Use Great Expectations in AWS Glue | Great Expectations](https://docs.greatexpectations.io/docs/0.15.50/deployment_patterns/how_to_use_great_expectations_in_aws_glue/)  
My problem is to understand what to set for datasource\_name as when I use the name provided by the tutorial it is not working. I got this error message : _DatasourceError: Cannot initialize datasource version-0.15.50 spark\_s3, error: The given datasource could not be retrieved from the DataContext; please confirm that your configuration is accurate._  
From the code provided it looks like this:  
expectation\_suite\_name = “version-0.15.50 suite\_name”  
suite = context\_gx.add\_expectation\_suite(expectation\_suite\_name)

```auto
batch_request = RuntimeBatchRequest(
    datasource_name="version-0.15.50 spark_s3",
    data_asset_name="version-0.15.50 datafile_name",
    batch_identifiers={"runtime_batch_identifier_name": "default_identifier"},
    data_connector_name="version-0.15.50 default_inferred_data_connector_name",
    runtime_parameters={"batch_data": df},
)

```

I understand that datasource is what kind of data such as in this case using pyspark, but what is datasource\_name. I don’t see in the yaml where it is set, it looks like this:

```auto
config_version: 3.0
datasources:
  spark_s3:
    module_name: great_expectations.datasource
    class_name: Datasource
    execution_engine:
      module_name: great_expectations.execution_engine
      class_name: SparkDFExecutionEngine
    data_connectors:
      default_runtime_data_connector_name:
        batch_identifiers:
          - runtime_batch_identifier_name
        module_name: great_expectations.datasource.data_connector
        class_name: RuntimeDataConnector

```

Is the datasource\_name by default or should i set it ? Can you help me understand that please? Actually if someone can explain me all parameters to set for RuntimeBatchRequest (datasource\_name, data\_asset\_name, batch\_identifiers, data\_connector\_name, data\_connector\_name) that would be awesome because it is not clear to me.

---

<div class="post-metadata">

**Author:** ![nevintan](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/nevintan/32/272_2.png) [@nevintan](https://discourse.greatexpectations.io/u/nevintan)\
**Post date:** [September 6, 2023, 3:49pm UTC](https://discourse.greatexpectations.io/t/help-understanding-datasource-name-config-for-runtimebatchrequest/1374/2 "2023-09-06T15:49:48Z")

</div>

Hi @anais, thanks for the question! Can you please verify which version of GX you are running this code with?

---

<div class="post-metadata">

**Author:** ![anais](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@anais](https://discourse.greatexpectations.io/u/anais)\
**Post date:** [September 7, 2023, 7:56am UTC](https://discourse.greatexpectations.io/t/help-understanding-datasource-name-config-for-runtimebatchrequest/1374/3 "2023-09-07T07:56:30Z")

</div>

I use the version 0.15.50 as for now it is the last version that support aws glue.  
I finally found that the datasource\_name has to be the one written in the yaml file, which is “spark\_s3” and not working as it was written in the tutorial “version-0.15.50 spark\_s3”, same for data\_asset\_name and data\_connector\_name I removed the mention of the version.  
I also need to add this parameter “force\_reuse\_spark\_context: true” under execution\_engine , otherwise it was not working.  
I also adapt the end of the script as the checkpoint part was not working with the providing code.  
At the end the data\_asset\_name is still not clear to me what is it use for as the data seems identified by the batch identifiers.

---

<div class="post-metadata">

**Author:** ![HaebichanGX](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/haebichangx/32/211_2.png) [@HaebichanGX](https://discourse.greatexpectations.io/u/HaebichanGX)\
**Post date:** [September 13, 2023, 3:31pm UTC](https://discourse.greatexpectations.io/t/help-understanding-datasource-name-config-for-runtimebatchrequest/1374/4 "2023-09-13T15:31:30Z")

</div>

Ah yes, so @anais our support for AWS Glue is only up to 0.15 and that version is quite old and deprecated. Our new version doesn’t fully support AWS Glue as it has not been tested on our end. Therefore, it’s likely that the tutorial will not work.

---

<div class="post-metadata">

**Author:** ![anais](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@anais](https://discourse.greatexpectations.io/u/anais)\
**Post date:** [September 14, 2023, 8:47am UTC](https://discourse.greatexpectations.io/t/help-understanding-datasource-name-config-for-runtimebatchrequest/1374/5 "2023-09-14T08:47:59Z")

</div>

Thank you for your answer, I finally manage to make it work.  
But I wonder if I should keep going with GX as you said the version is old and deprecated. Do you know if it will be integrated in the new version?

---

<div class="post-metadata">

**Author:** ![HaebichanGX](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/haebichangx/32/211_2.png) [@HaebichanGX](https://discourse.greatexpectations.io/u/HaebichanGX)\
**Post date:** [September 14, 2023, 5:58pm UTC](https://discourse.greatexpectations.io/t/help-understanding-datasource-name-config-for-runtimebatchrequest/1374/6 "2023-09-14T17:58:34Z")

</div>

Hey @anais can you share your solution here? So it can be helpful for our community. I don’t think there is a timeline for this request yet due to our internal prioritization.