I am woking with a spark_abs data source and am finding it hard to connect to my data stored in the delta format.
setup
- context: FileDataContext
- data-source: SparkAzureBlobStorageDatasource
- data-asset: add_directory_delta_asset
problem
When reading from an unpartitioned delta table i find that the reader attempts to read file by file which causes a partition fragment AnalysisException. Note that for this attempt the delta-table is written without any partitions.
pth=<SOME PATH FROM ROOT OF CONTAINER>
abs_name_starts_with=pth
data_directory=pth
I have yet been able to find any good documentation on the delta_asset (or it’s directory counterpart) thus I am left questioning the impact of some of the additional requirements such as the recursive file discoverry (does the delta asset not understand the partitioning set when writing?) version_as_of
, is that used for versioning the asset in case of changes to the data_directory?
Thank you kindly in advance for the help on this matter