Hi everyone,
I am currently trying to run a in memory dataframe through a checkpoint and afterwards create the html data docs to get some run statistics. The only problem is that my Asset Name will not show up in the generated docs.
The way I have setup the code flow right now is that I have added a spark Datasource and subsequently also added a dataframe_asset to this Datasource. I then create a spark dataframe with the data I would like to test. I build a batch_reqeust on the spark dataframe with the build_batch_request(dataframe=df) method. However, when I then run the batch_request through a checkpoint, my Datasource Name and Asset Name gets concatenated into the Batch ID?
I have tried altering the batch_request object, but no matter what I do the Asset Name in the data docs is still empty.
I have also checked out the Asset Name not shown topic. But since I create my batch_request with the build_batch_request(dataframe=df) method, I do not see a way to configure the batch_spec_passthrough options.
Does anybody have a suggestions?
Thanks in advance!