Hi I am trying to read a parquet file stored in azure blob with the following structure, but the batch definition only brings 16 records when running expectations (taking only one of those files as reference). I have used the add_batch_definition_whole_directory
from the docs to create the batch definition but no luck… Am I doing something wrong or is there a more appropriate way of reading this data?
The parquet files are all in one folder with this format:
part-00000-tid-1452779377459543239-207fab60-6c7a-4eec-b6b8-7d2002faf21a-11-1-c000.snappy.parquet
Here is the code I used to initialize it:
Thanks in advance!!