Great expectation is only returning 20 rows of failed records

GX Core Support
, , , ,
1

Hello Everyone,

I’m currently working on a project that uses the Snowflake connector with Great Expectations (version 0.17.23). I’ve set up some validation rules, and my data is failing on several of these. However, even though more than 1000 records fail, Great Expectations only returns a maximum of 20 failed records per rule.

After investigating with the Snowflake debugger, we discovered that Great Expectations is automatically adding a LIMIT 20 clause when fetching the failed records. I would like to disable this limit to retrieve details for all failed records rather than just 20.

Any guidance on how to remove this limit would be greatly appreciated!

2

hi there. where are you seeing this limit of 20? Data docs or in your validation results? The limit in Data docs is because Data docs is more meant to summarize for stakeholders and not so much as a debugging tool.

When it comes to your validation results you can set the level of detail returned in your Validation Results by specifying a value for the optional result_format parameter. More detail here.

3

so the limit is 200 for either unexpected_list (with “COMPLETE” result parameter) or partial_unexpected_list (with “SUMMARY”)?

5

There should be no limit in validation results when using COMPLETE

6

Hi again, I could not do that, the output was always 200 elements and I was sure there was a lot of more since out of 64000 objects, 63826 did not pass validation.
Maybe you could point me what am I doing wrong in my implementation?

import great_expectations as ge
import great_expectations.expectations as gex

context = ge.get_context(mode=“ephemeral”)
data_source = context.data_sources.add_spark(name=“my_data_source”)
data_asset = data_source.add_dataframe_asset(name=“my_data_asset”)
batch_definition = data_asset.add_batch_definition_whole_dataframe(“my_batch_definition”)
suite = ge.ExpectationSuite(name=“my_suite”)
suite = context.suites.add(suite)
suite.add_expectation(… )
suite.add_expectation etc. etc.
validation_definition = context.validation_definitions.add(ge.core.validation_definition.ValidationDefinition(name=“my_validation_definition”, data=batch_definition, suite=suite,))
results = validation_definition.run(batch_parameters={“dataframe”: my_df}, result_format={“result_format”: “COMPLETE”}

i tried also:
results = validation_definition.run(batch_parameters={“dataframe”: my_df}, result_format={“result_format”: “COMPLETE”, “partial_unexpected_count”: 1500}
but it didn’t help. Im also not aware of any parameter for customizing unexpected_list count

7

are you seeing they key unexpected_index_query returned in your results dictionary when using COMPLETE? That query can be used to retrieve all unexpected values without a limit

1 Like
8

I, too, am noticing this even when using the unexpected_index_list. it is limiting it to 20, but appears to only do so with SQL runs. For Pandas, the full unexpected_index_list shows up. we ran with 168 failure rows.

9

Hi,
I have a similar issue. I am using result_format="COMPLETE", include_unexpected_rows=True. Data source is Snowflake. The returned result shows 20 items for violated expectations, but it does not show the failed rows. The total number of violations is 856484, which is returned in the result, but we need all the rows with a violation.

result_format_complete =  {"result_format": "COMPLETE", "return_unexpected_index_query":False, "include_unexpected_rows": True}
expectation = gx.expectations.ExpectColumnValuesToNotBeInSet(
    column="L_SHIPMODE",
    value_set=['RAIL'],
        result_format=result_format_complete,
)

result:

{
  "success": false,
  "expectation_config": {
    "type": "expect_column_values_to_not_be_in_set",
    "kwargs": {
      "result_format": {
        "result_format": "COMPLETE",
        "return_unexpected_index_query": false,
        "include_unexpected_rows": true
      },
      "batch_id": "inergy-rttpc_lineitem",
      "column": "L_SHIPMODE",
      "value_set": [
        "RAIL"
      ]
    },
    "meta": {}
  },
  "result": {
    "element_count": 6001215,
    "unexpected_count": 856484,
    "unexpected_percent": 14.271843285068107,
    "partial_unexpected_list": [
      "RAIL",
      "RAIL",
      "RAIL",
      "RAIL",
      "RAIL",
      "RAIL",
      "RAIL",
      "RAIL",
      "RAIL",
      "RAIL",
      "RAIL",
      "RAIL",
      "RAIL",
      "RAIL",
      "RAIL",
      "RAIL",
      "RAIL",
      "RAIL",
      "RAIL",
      "RAIL"
    ],
    "missing_count": 0,
    "missing_percent": 0.0,
    "unexpected_percent_total": 14.271843285068107,
    "unexpected_percent_nonmissing": 14.271843285068107,
    "partial_unexpected_counts": [
      {
        "value": "RAIL",
        "count": 20
      }
    ]
  },
  "meta": {},
  "exception_info": {
    "raised_exception": false,
    "exception_traceback": null,
    "exception_message": null
  }
}