Hi I’m new to gx and I was curious about memory use / efficiency.
Say there’s a table with 2+ years of data being used as a data source or say a pandas df. With the expectation of expect_column_values_to_be_unique, do you have to rerun the expectation on the entire data source each time you want to verify there are no duplicates? Or is there a way to save resources by only having to run it on the new data added to the table after running the entire 2+ years worth only once? My hunch is you have to run it against the entire table as you need to check old records to see if any of the new records are duplicates.
I ask this because the batch / checkpoint workflow kinda confuses me.