Reading delta table data with supported 'DirectoryDeltaAsset' or 'DeltaAsset'

GX Core (OSS) Support
1

I am woking with a spark_abs data source and am finding it hard to connect to my data stored in the delta format.

setup

  • context: FileDataContext
  • data-source: SparkAzureBlobStorageDatasource
  • data-asset: add_directory_delta_asset

problem

When reading from an unpartitioned delta table i find that the reader attempts to read file by file which causes a partition fragment AnalysisException. Note that for this attempt the delta-table is written without any partitions.

pth=<SOME PATH FROM ROOT OF CONTAINER>

  • abs_name_starts_with=pth
  • data_directory=pth

I have yet been able to find any good documentation on the delta_asset (or it’s directory counterpart) thus I am left questioning the impact of some of the additional requirements such as the recursive file discoverry (does the delta asset not understand the partitioning set when writing?) version_as_of, is that used for versioning the asset in case of changes to the data_directory?

Thank you kindly in advance for the help on this matter