# GE in Databricks

**URL:** <https://discourse.greatexpectations.io/t/ge-in-databricks/114>\
**Category:** Archive\
**Created:** [May 11, 2020, 6:00pm UTC](https://discourse.greatexpectations.io/t/ge-in-databricks/114 "2020-05-11T18:00:00Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ychebaro](https://avatars.discourse-cdn.com/v4/letter/y/43a26b/32.png) [@ychebaro](https://discourse.greatexpectations.io/u/ychebaro)\
**Post date:** [May 11, 2020, 6:00pm UTC](https://discourse.greatexpectations.io/t/ge-in-databricks/114/1 "2020-05-11T18:00:00Z")

</div>

Hi all, I am trying to integrate GE with Databricks following [https://www.unsupervised-learnings.co.uk/post/setting-your-data-expectations-data-profiling-and-testing-with-the-great-expectations-library/](https://slack-redir.net/link?url=https%3A%2F%2Fwww.unsupervised-learnings.co.uk%2Fpost%2Fsetting-your-data-expectations-data-profiling-and-testing-with-the-great-expectations-library%2F) and I would like to save our own expectations without using the full profiling expectations to fit our needs.  
But I am running into trouble when saving the expectation suite:  
context.save\_expectation\_suite(discard\_failed\_expectations = False)  
save\_expectation\_suite\_usage\_statistics() got an unexpected keyword argument 'discard\_failed\_expectations’Am I missing something? Thank you!

---

<div class="post-metadata">

**Author:** ![eugene.mandel](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/eugene.mandel/32/22_2.png) [@eugene.mandel](https://discourse.greatexpectations.io/u/eugene.mandel)\
**Post date:** [May 11, 2020, 6:08pm UTC](https://discourse.greatexpectations.io/t/ge-in-databricks/114/2 "2020-05-11T18:08:12Z")

</div>

This blog post is indeed awesome!

Looks like a mistake sneaked in into this code snippet in a recent edit:

```
spark_data.expect_column_values_to_not_be_null("PROJECT_ID")
spark_data.expect_column_values_to_be_in_set("DELIVERY_GROUP_LATEST", set(['IP Legacy', 'IP AM SP&C']))

spark_data.validate()

context.save_expectation_suite(discard_failed_expectations = False)

```

It should be “spark\_data.save\_expectation\_suite(discard\_failed\_expectations = False)”, not context.

---

<div class="post-metadata">

**Author:** ![ychebaro](https://avatars.discourse-cdn.com/v4/letter/y/43a26b/32.png) [@ychebaro](https://discourse.greatexpectations.io/u/ychebaro)\
**Post date:** [May 11, 2020, 6:26pm UTC](https://discourse.greatexpectations.io/t/ge-in-databricks/114/3 "2020-05-11T18:26:40Z")

</div>

Thanks @eugene.mandel!  
I think things are getting mixed up in my config, here is the error I get

```auto
Unable to save config: filepath or data_context must be available.

```

I had to change some of the config options in the yml to make it work initially, but I am not sure I have the correct syntax:

```auto
datasources:
  spark:
    class_name: SparkDFDatasource
    batch_kwargs_generators:
      passthrough:
        class_name: DatabricksTableBatchKwargsGenerator
    data_asset_type:
      module_name:
      class_name: SparkDFDataset

```

Thanks for the help!

---

<div class="post-metadata">

**Author:** ![eugene.mandel](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.greatexpectations.io/eugene.mandel/32/22_2.png) [@eugene.mandel](https://discourse.greatexpectations.io/u/eugene.mandel)\
**Post date:** [May 11, 2020, 6:57pm UTC](https://discourse.greatexpectations.io/t/ge-in-databricks/114/4 "2020-05-11T18:57:01Z")

</div>

I see - “spark\_data. save\_expectation\_suite” can be called with these arguments if spark\_data is associated with a Data Context object. Usually, we obtain a data asset (in this case “spark\_data”) from Data Context, but to make it work in this code snippet, we can provide a reference to a DataContext when we create the spark\_data object:

instead of :  
`spark_data = SparkDFDataset(data)`

this:  
`spark_data = SparkDFDataset(data, data_context=context)`

---

<div class="post-metadata">

**Author:** ![ychebaro](https://avatars.discourse-cdn.com/v4/letter/y/43a26b/32.png) [@ychebaro](https://discourse.greatexpectations.io/u/ychebaro)\
**Post date:** [May 11, 2020, 8:29pm UTC](https://discourse.greatexpectations.io/t/ge-in-databricks/114/5 "2020-05-11T20:29:48Z")

</div>

Fantastic!  
I think I’m almost there…  
I have now an expectation json file with only the one I specified, but I can’t seem to define the output filename, its always saved in default.json

```auto
data = spark.read.parquet("/xxxx/business_classification_dim_parquet")
data_asset_name = "business_classification_dim_parquet"

context.create_expectation_suite(data_asset_name, True)

spark_data = SparkDFDataset(data, data_context=context)

# create expectations:
spark_data.expect_table_columns_to_match_ordered_list(["xxxx", "xxx"])

spark_data.validate()

# save expectations
spark_data.save_expectation_suite("business_classification_dim_parquet")

```

Thank you so much!

---

<div class="post-metadata">

**Author:** ![ychebaro](https://avatars.discourse-cdn.com/v4/letter/y/43a26b/32.png) [@ychebaro](https://discourse.greatexpectations.io/u/ychebaro)\
**Post date:** [May 11, 2020, 9:46pm UTC](https://discourse.greatexpectations.io/t/ge-in-databricks/114/6 "2020-05-11T21:46:05Z")

</div>

Ok I think I figured it out, this seems to work now:

```auto
spark_data.save_expectation_suite(filepath="/dbfs/ge_er_test/expectations/business_classification11-05-2020_expectations.json")

```

Now one last question, is it possible to read delta tables instead of parquet? I added

```auto
reader_method: delta

```

in the config file, but it doesn’t seem that spark.read.parquet can be replaced by spark.read.delta?

Thanks again @eugene.mandel !