When using the example dataset, I am trying to run this line of code on the “congestion_surcharge” column which is of type float64 and has some null values in the column: validator.expect_column_values_to_not_be_null("congestion_surcharge")
However, it doesn’t show up in the data doc html page, but the column " total_amount" shows up without any problems. See picture below.
#create expectations
validator.expect_column_values_to_not_be_null("pickup_datetime")
validator.expect_column_values_to_not_be_null("congestion_surcharge")
validator.expect_column_values_to_not_be_null("total_amount")
validator.expect_column_values_to_be_between(
"passenger_count", min_value=1, max_value=6
)
validator.save_expectation_suite()
Code Snippet
Data columns (total 18 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 vendor_id 10000 non-null int64
1 pickup_datetime 10000 non-null object
2 dropoff_datetime 10000 non-null object
3 passenger_count 10000 non-null int64
4 trip_distance 10000 non-null float64
5 rate_code_id 10000 non-null int64
6 store_and_fwd_flag 10000 non-null object
7 pickup_location_id 10000 non-null int64
8 dropoff_location_id 10000 non-null int64
9 payment_type 10000 non-null int64
10 fare_amount 10000 non-null float64
11 extra 10000 non-null float64
12 mta_tax 10000 non-null float64
13 tip_amount 10000 non-null float64
14 tolls_amount 10000 non-null float64
15 improvement_surcharge 10000 non-null float64
16 total_amount 10000 non-null float64
17 congestion_surcharge 3673 non-null float64
Column Data Types
For other int64, object (with no null values) and float64 (with no null values) types the same expectation is working without any issues. It seems like with this expectation, when a column contains some null values it will not show up in the data docs html page. Has anyone else encountered this problem? Thank you in advance!