# \#databricks

**URL:** https://discourse.greatexpectations.io/tag/databricks/4.md

[Latest](https://discourse.greatexpectations.io/latest.md) · [Categories](https://discourse.greatexpectations.io/categories.md) · [Tags](https://discourse.greatexpectations.io/tags.md)

---

## [GX validation Result returns broken query "unexpected\_index\_query"](https://discourse.greatexpectations.io/t/gx-validation-result-returns-broken-query-unexpected-index-query/2039)

<div class="topic-metadata">

**Author:** [@Jyoti\_Thakkar](https://discourse.greatexpectations.io/u/Jyoti_Thakkar)\
**Replies:** 3\
**Last updated:** [October 27, 2025, 8:26am UTC](https://discourse.greatexpectations.io/t/gx-validation-result-returns-broken-query-unexpected-index-query/2039 "2025-10-27T08:26:11Z")

</div>

Using GX 1.2.3 version we have set result\_format as “COMPLETE” and unexpected\_index\_query to true, we want to use unexpected\_index\_query to get the error records from dataframe. It seems that it is returning broken quer…

---

## [Is unexpected\_index\_query will be generated in result format "Summary"](https://discourse.greatexpectations.io/t/is-unexpected-index-query-will-be-generated-in-result-format-summary/2304)

<div class="topic-metadata">

**Author:** [@Swarup](https://discourse.greatexpectations.io/u/Swarup)\
**Replies:** 0\
**Last updated:** [October 11, 2025, 3:47pm UTC](https://discourse.greatexpectations.io/t/is-unexpected-index-query-will-be-generated-in-result-format-summary/2304 "2025-10-11T15:47:29Z")

</div>

Is yes can someone help me to get that?

---

## [Group By Expectations. 1.6.4](https://discourse.greatexpectations.io/t/group-by-expectations-1-6-4/2300)

<div class="topic-metadata">

**Author:** [@filiphjartstrom](https://discourse.greatexpectations.io/u/filiphjartstrom)\
**Replies:** 0\
**Last updated:** [October 7, 2025, 8:28am UTC](https://discourse.greatexpectations.io/t/group-by-expectations-1-6-4/2300 "2025-10-07T08:28:20Z")

</div>

Hello! I have searched the docs and git and cannot figure something out: Lets say i want to use the ExpectColumnSumToBeBetween expectation. How can I group by another column? For example: Group purchases by country and…

---

## [Keep getting batch\_definition name already exists - In Databricks](https://discourse.greatexpectations.io/t/keep-getting-batch-definition-name-already-exists-in-databricks/2285)

<div class="topic-metadata">

**Author:** [@Harpreet](https://discourse.greatexpectations.io/u/Harpreet)\
**Replies:** 1\
**Last updated:** [August 23, 2025, 2:09am UTC](https://discourse.greatexpectations.io/t/keep-getting-batch-definition-name-already-exists-in-databricks/2285 "2025-08-23T02:09:30Z")

</div>

Hello everyone, I am new to GX core and I am currently tryint to setup in Databricks notebook. On my first instance I was able to script and run an expectation but from the next day while running the same script I kee…

---

## [Unable to run ‘ExpectColumnPairValuesToBeEqual’ with spark on Databricks](https://discourse.greatexpectations.io/t/unable-to-run-expectcolumnpairvaluestobeequal-with-spark-on-databricks/2062)

<div class="topic-metadata">

**Author:** [@motaslimi](https://discourse.greatexpectations.io/u/motaslimi)\
**Replies:** 9\
**Last updated:** [July 28, 2025, 12:12pm UTC](https://discourse.greatexpectations.io/t/unable-to-run-expectcolumnpairvaluestobeequal-with-spark-on-databricks/2062 "2025-07-28T12:12:38Z")

</div>

When I’m trying to validate any expectation with parameters column\_A, column\_B, I get the following error: \[CANNOT\_RESOLVE\_DATAFRAME\_COLUMN\] Cannot resolve dataframe column "col\_1". It’s probably because of illegal refe…

---

## [Does GX Support Unity Catalog enabled Workspace](https://discourse.greatexpectations.io/t/does-gx-support-unity-catalog-enabled-workspace/1395)

<div class="topic-metadata">

**Author:** [@sanath](https://discourse.greatexpectations.io/u/sanath)\
**Replies:** 2\
**Last updated:** [July 3, 2025, 3:21pm UTC](https://discourse.greatexpectations.io/t/does-gx-support-unity-catalog-enabled-workspace/1395 "2025-07-03T15:21:28Z")

</div>

I am exploring GX for one of the projects that I am working on and I am using Databricks in azure hosted environment I have my workspace integrated with the Unity catalog. In this situation how do I use GX as mounting is…

---

## [Unable to load context](https://discourse.greatexpectations.io/t/unable-to-load-context/2239)

<div class="topic-metadata">

**Author:** [@Bhavana](https://discourse.greatexpectations.io/u/Bhavana)\
**Replies:** 0\
**Last updated:** [July 1, 2025, 12:14pm UTC](https://discourse.greatexpectations.io/t/unable-to-load-context/2239 "2025-07-01T12:14:47Z")

</div>

i have a data context created at a custom location using get\_context(mode = ‘file’, context\_root\_dir=path) function. after couple of days, when i try to load the existing context using get\_context(mode = ‘file’, context…

---

## [Data partially read when using: add\_batch\_definition\_whole\_directory()](https://discourse.greatexpectations.io/t/data-partially-read-when-using-add-batch-definition-whole-directory/2068)

<div class="topic-metadata">

**Author:** [@8toz](https://discourse.greatexpectations.io/u/8toz)\
**Replies:** 2\
**Last updated:** [June 18, 2025, 5:09pm UTC](https://discourse.greatexpectations.io/t/data-partially-read-when-using-add-batch-definition-whole-directory/2068 "2025-06-18T17:09:00Z")

</div>

Hi I am trying to read a parquet file stored in azure blob with the following structure, but the batch definition only brings 16 records when running expectations (taking only one of those files as reference). I have use…

---

## [Columns with Dots: The column in BatchData does not exist](https://discourse.greatexpectations.io/t/columns-with-dots-the-column-in-batchdata-does-not-exist/2189)

<div class="topic-metadata">

**Author:** [@Anna](https://discourse.greatexpectations.io/u/Anna)\
**Replies:** 1\
**Last updated:** [June 13, 2025, 12:05pm UTC](https://discourse.greatexpectations.io/t/columns-with-dots-the-column-in-batchdata-does-not-exist/2189 "2025-06-13T12:05:24Z")

</div>

Dear GX Community, I am using GX version 1.3.10 with spark/Databricks and when I evaluate an Expectation for a column name which include a dot, e.g. “Te.st” I get the error that GX cannot find the column. Is there any w…

---

## [ExpectColumnValuesToBeOfType is not working for databricks/spark](https://discourse.greatexpectations.io/t/expectcolumnvaluestobeoftype-is-not-working-for-databricks-spark/2115)

<div class="topic-metadata">

**Author:** [@mahfooz.iiitian](https://discourse.greatexpectations.io/u/mahfooz.iiitian)\
**Replies:** 2\
**Last updated:** [April 25, 2025, 2:07pm UTC](https://discourse.greatexpectations.io/t/expectcolumnvaluestobeoftype-is-not-working-for-databricks-spark/2115 "2025-04-25T14:07:17Z")

</div>

When we are running below expectation, it is throwing below error. expectation = gx.expectations.ExpectColumnValuesToBeOfType( column="test2", type\_="NUMBER", result\_format= "COMPLETE", ) Exception thrown: …

---

## [Valid Value Check For A Column Referencing Another Table](https://discourse.greatexpectations.io/t/valid-value-check-for-a-column-referencing-another-table/2117)

<div class="topic-metadata">

**Author:** [@sujeet\_juneja](https://discourse.greatexpectations.io/u/sujeet_juneja)\
**Replies:** 0\
**Last updated:** [March 5, 2025, 6:41pm UTC](https://discourse.greatexpectations.io/t/valid-value-check-for-a-column-referencing-another-table/2117 "2025-03-05T18:41:48Z")

</div>

Which check should I write where I want to check the value of a column should be present in my reference table?

---

## [One Common Docs For All The Expectations And Validations](https://discourse.greatexpectations.io/t/one-common-docs-for-all-the-expectations-and-validations/2113)

<div class="topic-metadata">

**Author:** [@sujeet\_juneja](https://discourse.greatexpectations.io/u/sujeet_juneja)\
**Replies:** 0\
**Last updated:** [March 3, 2025, 7:15pm UTC](https://discourse.greatexpectations.io/t/one-common-docs-for-all-the-expectations-and-validations/2113 "2025-03-03T19:15:02Z")

</div>

I want to keep one docs for all the different expectations that is run with it’s multiple validation also.

---

## [Metric/Evaluation Parameter store using Databricks delta tables](https://discourse.greatexpectations.io/t/metric-evaluation-parameter-store-using-databricks-delta-tables/1470)

<div class="topic-metadata">

**Author:** [@Tim](https://discourse.greatexpectations.io/u/Tim)\
**Replies:** 4\
**Last updated:** [February 20, 2025, 3:43pm UTC](https://discourse.greatexpectations.io/t/metric-evaluation-parameter-store-using-databricks-delta-tables/1470 "2025-02-20T15:43:56Z")

</div>

Hi GX people! I am currently using great expectations in Databricks. I would like to setup an Evaluation Parameter Store and/or Metric Store using the Databricks native delta tables in Unity Catalog Database. Does anyon…

---

## [Data Context](https://discourse.greatexpectations.io/t/data-context/2086)

<div class="topic-metadata">

**Author:** [@Anna](https://discourse.greatexpectations.io/u/Anna)\
**Replies:** 3\
**Last updated:** [February 12, 2025, 6:30pm UTC](https://discourse.greatexpectations.io/t/data-context/2086 "2025-02-12T18:30:10Z")

</div>

Hi Folks, I am new to great expectations and I have some questions to which I couldn’t find a clear answer so far. I am using GX on Databricks and I would like to validate incoming data. For each dataset which I want to…

---

## [How to Create Custom Expectation of type Multicolumn Map Expectation](https://discourse.greatexpectations.io/t/how-to-create-custom-expectation-of-type-multicolumn-map-expectation/2079)

<div class="topic-metadata">

**Author:** [@Jyoti\_Thakkar](https://discourse.greatexpectations.io/u/Jyoti_Thakkar)\
**Replies:** 1\
**Last updated:** [January 30, 2025, 10:48pm UTC](https://discourse.greatexpectations.io/t/how-to-create-custom-expectation-of-type-multicolumn-map-expectation/2079 "2025-01-30T22:48:27Z")

</div>

we are using GX 1.3.4 version in databricks and have a requirement where we have to check multiple column values are Null and if multiple column values are null then fail that row. We are looking for sample code snippet.…

---

## [How to create custom expectations using GX Core 1.2.0 version?](https://discourse.greatexpectations.io/t/how-to-create-custom-expectations-using-gx-core-1-2-0-version/1964)

<div class="topic-metadata">

**Author:** [@Dinesh](https://discourse.greatexpectations.io/u/Dinesh)\
**Replies:** 2\
**Last updated:** [January 24, 2025, 10:04am UTC](https://discourse.greatexpectations.io/t/how-to-create-custom-expectations-using-gx-core-1-2-0-version/1964 "2025-01-24T10:04:02Z")

</div>

Hi Team, We are trying to create a automation framework for testing data which covers business requirements related to Data migration and Data reconciliation. We have already created basic level framework using great ex…

---

## [Checkpoint with "result\_format": "COMPLETE" json that includes validation result without saving all results to GX context](https://discourse.greatexpectations.io/t/checkpoint-with-result-format-complete-json-that-includes-validation-result-without-saving-all-results-to-gx-context/1859)

<div class="topic-metadata">

**Author:** [@cary](https://discourse.greatexpectations.io/u/cary)\
**Replies:** 4\
**Last updated:** [December 23, 2024, 2:40pm UTC](https://discourse.greatexpectations.io/t/checkpoint-with-result-format-complete-json-that-includes-validation-result-without-saving-all-results-to-gx-context/1859 "2024-12-23T14:40:40Z")

</div>

Hello, I am using a checkpoint with “result\_format”: “COMPLETE” option in order to use the complete JSON for custom python flagging row by row issues with a unique identifier (in this case point\_id). The issue I am hav…

---

## [GX-Databricks:Datasource-Data asset - Validator](https://discourse.greatexpectations.io/t/gx-databricks-datasource-data-asset-validator/1738)

<div class="topic-metadata">

**Author:** [@kat](https://discourse.greatexpectations.io/u/kat)\
**Replies:** 8\
**Last updated:** [December 19, 2024, 3:28pm UTC](https://discourse.greatexpectations.io/t/gx-databricks-datasource-data-asset-validator/1738 "2024-12-19T15:28:45Z")

</div>

Hello, I use the notebooks produced from CLI and try to adapt them in order to run in Databricks so I run the following: import great\_expectations as gx import pandas as pd import great\_expectations.jupyter\_ux from …

---

## [Updating expectation definition](https://discourse.greatexpectations.io/t/updating-expectation-definition/2035)

<div class="topic-metadata">

**Author:** [@andy70](https://discourse.greatexpectations.io/u/andy70)\
**Replies:** 0\
**Last updated:** [December 19, 2024, 10:29am UTC](https://discourse.greatexpectations.io/t/updating-expectation-definition/2035 "2024-12-19T10:29:44Z")

</div>

hello I’m on ge 1.2.6 and working on databricks with a spark dataframe I have defined some expectations and then added them to an expectation suite: age\_expectation = gx.expectations.ExpectColumnMinToBeBetween( colum…

---

## [Migration From V0 to V1](https://discourse.greatexpectations.io/t/migration-from-v0-to-v1/2028)

<div class="topic-metadata">

**Author:** [@Jyoti\_Thakkar](https://discourse.greatexpectations.io/u/Jyoti_Thakkar)\
**Replies:** 2\
**Last updated:** [December 16, 2024, 6:05pm UTC](https://discourse.greatexpectations.io/t/migration-from-v0-to-v1/2028 "2024-12-16T18:05:16Z")

</div>

Hi GX Community, I am using GX 0.15.43 and want to migrate to GX CoreV1. We are using GX in databricks and instantiating a Data Context without a yml file. Below is the sample code import great\_expectations as ge fro…

---

## [Validate different dataframes with respective expectation suites using checkpoint](https://discourse.greatexpectations.io/t/validate-different-dataframes-with-respective-expectation-suites-using-checkpoint/1876)

<div class="topic-metadata">

**Author:** [@psof](https://discourse.greatexpectations.io/u/psof)\
**Replies:** 4\
**Last updated:** [October 10, 2024, 2:08pm UTC](https://discourse.greatexpectations.io/t/validate-different-dataframes-with-respective-expectation-suites-using-checkpoint/1876 "2024-10-10T14:08:26Z")

</div>

I am using Databricks with great expectations 1.0.1 version. I want with a single checkpoint to validate two different data assets with their respective expectations suites. I have the following: batch\_parameters = {"d…

---

## [I am looking for immediate use of GE library for project. which version could be use 0.17.x or 0.18.x?](https://discourse.greatexpectations.io/t/i-am-looking-for-immediate-use-of-ge-library-for-project-which-version-could-be-use-0-17-x-or-0-18-x/1917)

<div class="topic-metadata">

**Author:** [@shivakrishna9](https://discourse.greatexpectations.io/u/shivakrishna9)\
**Replies:** 0\
**Last updated:** [September 25, 2024, 9:57am UTC](https://discourse.greatexpectations.io/t/i-am-looking-for-immediate-use-of-ge-library-for-project-which-version-could-be-use-0-17-x-or-0-18-x/1917 "2024-09-25T09:57:30Z")

</div>

We have good number of tables which need lot of monitoring and I see GE could be the right usecase. Please help with the library which could be used in production environments and would need less refactor when we try to …

---

## [GX 1.0 and Databricks](https://discourse.greatexpectations.io/t/gx-1-0-and-databricks/1861)

<div class="topic-metadata">

**Author:** [@ToivoMattila](https://discourse.greatexpectations.io/u/ToivoMattila)\
**Replies:** 2\
**Last updated:** [September 3, 2024, 1:46pm UTC](https://discourse.greatexpectations.io/t/gx-1-0-and-databricks/1861 "2024-09-03T13:46:07Z")

</div>

I’m trying out the new 1.0 release on Databricks and going through the related documentation. What’s the recommended approach for working with GX 1.0 in Databricks and validating data in the Unity Catalog tables?

---

## [ModuleNotFoundError: No module named 'great\_expectations.dataset'](https://discourse.greatexpectations.io/t/modulenotfounderror-no-module-named-great-expectations-dataset/1852)

<div class="topic-metadata">

**Author:** [@techzy](https://discourse.greatexpectations.io/u/techzy)\
**Replies:** 5\
**Last updated:** [August 23, 2024, 3:00pm UTC](https://discourse.greatexpectations.io/t/modulenotfounderror-no-module-named-great-expectations-dataset/1852 "2024-08-23T15:00:57Z")

</div>

Hi everyone, I’m encountering a ‘ModuleNotFoundError: No module named ‘great\_expectations.dataset’’ error when attempting to import the dataset module in Great Expectations. This issue has arisen recently, as the code w…

---

## [SQL Data Asset throwing "NoneType object is not iterable" while validating in azure databricks](https://discourse.greatexpectations.io/t/sql-data-asset-throwing-nonetype-object-is-not-iterable-while-validating-in-azure-databricks/1737)

<div class="topic-metadata">

**Author:** [@Dinesh](https://discourse.greatexpectations.io/u/Dinesh)\
**Replies:** 4\
**Last updated:** [May 24, 2024, 10:04am UTC](https://discourse.greatexpectations.io/t/sql-data-asset-throwing-nonetype-object-is-not-iterable-while-validating-in-azure-databricks/1737 "2024-05-24T10:04:37Z")

</div>

Hi Team, I am using Azure databricks and great expectations to perform data testing on MS SQL Server currently. Recently I started implementing SQL Data Asset in place of Inmemory data asset to perform our validations. …

---

## [Expect\_column\_value\_lengths\_to\_be\_between: ERROR SQLSTATE 42704](https://discourse.greatexpectations.io/t/expect-column-value-lengths-to-be-between-error-sqlstate-42704/1725)

<div class="topic-metadata">

**Author:** [@kat](https://discourse.greatexpectations.io/u/kat)\
**Replies:** 1\
**Last updated:** [May 6, 2024, 3:27pm UTC](https://discourse.greatexpectations.io/t/expect-column-value-lengths-to-be-between-error-sqlstate-42704/1725 "2024-05-06T15:27:56Z")

</div>

Hello, I am trying to run the expectation in Azure Databricks environmment: validator.expect\_column\_value\_lengths\_to\_be\_between(column=“public\_remarks”, min\_value=1, max\_value=2000, result\_format={“result\_format”: “COM…

---

## [Publish great expectation result Azure DevOps Pipeline](https://discourse.greatexpectations.io/t/publish-great-expectation-result-azure-devops-pipeline/1676)

<div class="topic-metadata">

**Author:** [@mrajib74](https://discourse.greatexpectations.io/u/mrajib74)\
**Replies:** 5\
**Last updated:** [April 11, 2024, 8:37am UTC](https://discourse.greatexpectations.io/t/publish-great-expectation-result-azure-devops-pipeline/1676 "2024-04-11T08:37:55Z")

</div>

Hi, The customer wants to publish great expectation result which is stored in darabricks files system( Mounted to ADLS gen2 Storage location).in Azure devops cd pipeline .Please let me know how this can be achieved? Re…

---

## [Data Asset Name not showing in Data Docs](https://discourse.greatexpectations.io/t/data-asset-name-not-showing-in-data-docs/1583)

<div class="topic-metadata">

**Author:** [@Marques\_B](https://discourse.greatexpectations.io/u/Marques_B)\
**Replies:** 1\
**Last updated:** [March 7, 2024, 9:40pm UTC](https://discourse.greatexpectations.io/t/data-asset-name-not-showing-in-data-docs/1583 "2024-03-07T21:40:32Z")

</div>

Hi everyone, I am currently trying to run a in memory dataframe through a checkpoint and afterwards create the html data docs to get some run statistics. The only problem is that my Asset Name will not show up in the ge…

---

## [Issues with configuring metadata store connection to Azure Data Lake Storage Gen2](https://discourse.greatexpectations.io/t/issues-with-configuring-metadata-store-connection-to-azure-data-lake-storage-gen2/1572)

<div class="topic-metadata">

**Author:** [@Arni](https://discourse.greatexpectations.io/u/Arni)\
**Replies:** 0\
**Last updated:** [January 11, 2024, 4:43pm UTC](https://discourse.greatexpectations.io/t/issues-with-configuring-metadata-store-connection-to-azure-data-lake-storage-gen2/1572 "2024-01-11T16:43:53Z")

</div>

Hello, I’m trying to configure a Great Expectation context in a notebook in Databricks with metadata stores saved in an Azure Data Lake Storage Gen2 container but I’m getting connection errors. For this configuration pr…

---

## [Problem: validator does not work on a shared high concurrency cluster](https://discourse.greatexpectations.io/t/problem-validator-does-not-work-on-a-shared-high-concurrency-cluster/1494)

<div class="topic-metadata">

**Author:** [@Nadya](https://discourse.greatexpectations.io/u/Nadya)\
**Replies:** 3\
**Last updated:** [December 15, 2023, 10:44pm UTC](https://discourse.greatexpectations.io/t/problem-validator-does-not-work-on-a-shared-high-concurrency-cluster/1494 "2023-12-15T22:44:20Z")

</div>

Hello, I’m running GX on a shared cluster that has not enabled credential-passthrough for user-level data access. This code snippet: validator = context.get\_validator( batch\_request=batch\_request, expectation\_suite\_n…

[Next page](https://discourse.greatexpectations.io/tag/databricks/4.md?match_all_tags=true&page=1&tags%5B%5D=databricks)