# Is it possible to see the number of record in pyspark dataframe that didn't pass the validation?

**URL:** <https://discourse.greatexpectations.io/t/is-it-possible-to-see-the-number-of-record-in-pyspark-dataframe-that-didnt-pass-the-validation/1955>\
**Category:** GX Core Support\
**Tags:** how-to, help-wanted\
**Created:** [October 18, 2024, 10:53am UTC](https://discourse.greatexpectations.io/t/is-it-possible-to-see-the-number-of-record-in-pyspark-dataframe-that-didnt-pass-the-validation/1955 "2024-10-18T10:53:44Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![woodbine](https://avatars.discourse-cdn.com/v4/letter/w/a183cd/32.png) [@woodbine](https://discourse.greatexpectations.io/u/woodbine)\
**Post date:** [October 18, 2024, 10:53am UTC](https://discourse.greatexpectations.io/t/is-it-possible-to-see-the-number-of-record-in-pyspark-dataframe-that-didnt-pass-the-validation/1955/1 "2024-10-18T10:53:44Z")

</div>

Im validating big dataset in aws glue using pyspark dataframe. I have created properly data\_source with .add\_spark(), data\_asset, batch definition, ExpectationSuite, validation\_definition  
When I’m reading the result in “SUMMARY” format i can see the results of validation. Eg. for one of my created expectation I can see:  
“success”: false,  
“expectation\_config”: {  
“type”: “expect\_column\_value\_lengths\_to\_equal”,  
“kwargs”: {  
“batch\_id”: “my\_data\_source-my\_data\_asset”,  
“column”: “some\_column”,  
“value”: 9.0  
},  
“meta”: {},  
“id”: “some\_id\_12323-sdf324-etc”  
},  
“result”: {  
“element\_count”: 64060,  
“unexpected\_count”: 63796,  
“unexpected\_percent”: 99,58788635654075,  
“partial\_unexpected\_list”: [  
“12345678”,  
“12345678”,  
“12345678”,  
“12345678”,  
“12345678”,  
“12345678”,  
“12345678”,  
],  
“missing\_count”: 0,  
“missing\_percent”: 0.0,  
“unexpected\_percent\_total”: 99.58788635654075,  
“unexpected\_percent\_nonmissing”: 99.58788635654075,  
“partial\_unexpected\_counts”: [  
{  
“value”: “12345678”,  
“count”: 20  
}  
]  
},  
“meta”: {},  
“exception\_info”: {  
“raised\_exception”: false,  
“exception\_traceback”: null,  
“exception\_message”: null,  
},

I was wondering if there is a possibility to show the number of record (in the order that record was presented in dataframe) that didn’t pass validation?  
Thanks, Kuba

---

<div class="post-metadata">

**Author:** ![adeola](https://avatars.discourse-cdn.com/v4/letter/a/b487fb/32.png) [@adeola](https://discourse.greatexpectations.io/u/adeola)\
**Post date:** [October 23, 2024, 3:05pm UTC](https://discourse.greatexpectations.io/t/is-it-possible-to-see-the-number-of-record-in-pyspark-dataframe-that-didnt-pass-the-validation/1955/2 "2024-10-23T15:05:18Z")

</div>

hey there @woodbine welcome to our community

can you try adding `unexpected_index_column_names` to your result format? If your DataFrame has a unique identifier column (like an ID or record number), specify that column in `unexpected_index_column_names`. This will include the failing record indices in the validation output.

```auto
result_format = {
   "result_format": "SUMMARY",
   "unexpected_index_column_names": ["id_column"] 
}

```

replace “id\_column” with your unique identifier column