How to actually create a custom Query Expectations

GX OSS Support
,
1

After trying to replicate the tutorials on creating a custom Query expectation, copying the already implemented ones, I find myself unable to import any Query Expectation into my GX suite. Even just adding the already shipped ones in my GX suite (like “expect_queried_column_value_frequency_to_meet_threshold”) just yields the Error

‘ValueError: Unable to determine “domain_type” of this “ExpectationConfiguration” object from “kwargs” and heuristics.’

I am using GX 18.3,Python 3.11 with Pyspark on a small data frame. My Code otherwise works if I don’t use the any Query Metrics.

2

Hi @mant to get custom expectations to get imported, you need to follow few steps.

  1. Create your context using this code:
import great_expectations as gx
from great_expectations.core.expectation_configuration import ExpectationConfiguration
import pandas as pd
import os 
from great_expectations.data_context import FileDataContext

base = os.getcwd()

path_to_empty_folder = base + "/my_gx_project/"

context = FileDataContext.create(project_root_dir=path_to_empty_folder)
  1. Put your custom expectations code inside great_expectations → plugins → expectations
  2. Then run from my_gx_project.great_expectations.plugins.expectations import your_custom_expectation. This is if your GX notebook is in the same root as the project folder (I called it “my_gx_project”).

Remember that context needs to be called before importing custom expectations.

3

Thanks for your answer, but this is not my problem.

I cannot add QUERY expectations to my suite. I followed the tutorial on creating CUSTOM QUERY EXPECTATIONS. I add them to my plugins folder and then import them. Then I get the error

Traceback (most recent call last):
  File "/Users/christoph/PycharmProjects/great_expectations_test/gx_cli_test_pyspark.py", line 33, in <module>
    my_suite.show_expectations_by_expectation_type()
  File "/Users/christoph/PycharmProjects/great_expectations_test/.venv/lib/python3.11/site-packages/great_expectations/core/expectation_suite.py", line 798, in show_expectations_by_expectation_type
    self.get_grouped_and_ordered_expectations_by_expectation_type()
  File "/Users/christoph/PycharmProjects/great_expectations_test/.venv/lib/python3.11/site-packages/great_expectations/core/expectation_suite.py", line 861, in get_grouped_and_ordered_expectations_by_expectation_type
    self.get_table_expectations(),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/christoph/PycharmProjects/great_expectations_test/.venv/lib/python3.11/site-packages/great_expectations/core/expectation_suite.py", line 885, in get_table_expectations
    expectation_configurations: List[ExpectationConfiguration] = list(
                                                                 ^^^^^
  File "/Users/christoph/PycharmProjects/great_expectations_test/.venv/lib/python3.11/site-packages/great_expectations/core/expectation_suite.py", line 887, in <lambda>
    lambda element: element.get_domain_type() == MetricDomainTypes.TABLE,
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/christoph/PycharmProjects/great_expectations_test/.venv/lib/python3.11/site-packages/great_expectations/core/expectation_configuration.py", line 1514, in get_domain_type
    raise ValueError(
ValueError: Unable to determine "domain_type" of this "ExpectationConfiguration" object from "kwargs" and heuristics.```

This happens with ALL QUERY expectations in Great Expectations. 
This is what I add to my Suite.json file

{
“expectation_type”: “expect_queried_column_list_to_be_unique”,
“kwargs”: {
“template_dict”: {
“column_list”: “id”
}
},
“meta”: {
“notes”: {
“format”: “markdown”,
“content”: “Some clever comment about this expectation. Markdown Supported”
}
}
}