Expect_column_values_to_not_be_null column not showing up when column contains some null values

When using the example dataset, I am trying to run this line of code on the “congestion_surcharge” column which is of type float64 and has some null values in the column: validator.expect_column_values_to_not_be_null("congestion_surcharge")

However, it doesn’t show up in the data doc html page, but the column " total_amount" shows up without any problems. See picture below.

#create expectations
    "passenger_count", min_value=1, max_value=6

Code Snippet

Data columns (total 18 columns):
 #   Column                 Non-Null Count  Dtype  
---  ------                 --------------  -----  
 0   vendor_id              10000 non-null  int64  
 1   pickup_datetime        10000 non-null  object 
 2   dropoff_datetime       10000 non-null  object 
 3   passenger_count        10000 non-null  int64  
 4   trip_distance          10000 non-null  float64
 5   rate_code_id           10000 non-null  int64  
 6   store_and_fwd_flag     10000 non-null  object 
 7   pickup_location_id     10000 non-null  int64  
 8   dropoff_location_id    10000 non-null  int64  
 9   payment_type           10000 non-null  int64  
 10  fare_amount            10000 non-null  float64
 11  extra                  10000 non-null  float64
 12  mta_tax                10000 non-null  float64
 13  tip_amount             10000 non-null  float64
 14  tolls_amount           10000 non-null  float64
 15  improvement_surcharge  10000 non-null  float64
 16  total_amount           10000 non-null  float64
 17  congestion_surcharge   3673 non-null   float64

Column Data Types

For other int64, object (with no null values) and float64 (with no null values) types the same expectation is working without any issues. It seems like with this expectation, when a column contains some null values it will not show up in the data docs html page. Has anyone else encountered this problem? Thank you in advance!

Hi @vavass18 this is because you need to run validator.expect_column_values_to_not_be_null(“congestion_surcharge”, discard_failed_expectations = False).

It’s not showing up because congestion_surcharge failed in expectations.

1 Like

Hi @HaebichanGX,

The above solution didn’t work. However adding the option to the “save_expectation_suite” function worked (see code below)!

validator.save_expectation_suite(discard_failed_expectations = False)

Thank you for your help!