ExpectTableRowCount under the hood

GX Core Support
1

Hello,

I went through docs, but couldn’t find how batch request retrieve data eg for ExpectTableRowCount.
Assuming I have huge table, will it retrieve all records (and oh no store in memory) or it will just request row count from the DB

If I missed it in docs, could you, please, refer it to me?

Thanks

2

GX will just request row count from the DB.

When validating data in a Database, GX will leverage the database to do the heavy lifting. GX runs SQL queries and only uses the results from those queries.

GX will only load the data to memory when using pandas DataFrames (and maybe Spark, don’t remember that right now).

I couldn’t find any documentation about this either.

@mollie.pettit You might be interested.

However, I checked this in the sources here.
The row count metric is calculated just by running sqlalchemy.func.count().