Publish great expectation result Azure DevOps Pipeline

Hi,
The customer wants to publish great expectation result which is stored in darabricks files system( Mounted to ADLS gen2 Storage location).in Azure devops cd pipeline .Please let me know how this can be achieved?

Regards,
Rajib

Hello @mrajib74 ,
have you seen this post:
Data Docs in Azure ADLS $web subdirectory not working - GX OSS Support - Great Expectations
?
There it is described how it works to get data docs results written into ADLS Gen2 $web container.
It’s not a complete solution for this topic, but roughly descriped how ti works.

To get the connection string for authentication to ADLS I ordered a secret on the container which I can use for the connection_string variable.

If you have more detailed questions, please ask.

What do you mean with in Azure devopy cd pipeline?
What do you want to put in there regarding GX?
Eg. put the expectations jsons in the main branch or anything like that?

Greetings, Holger

Thanks @hdamczy for your response.
I will check the topic that you mentioned.
The great expectation test cases triggered from Azure Devops CD pipeline. If acceptance test is green then artifacts will be deployed in staging environment. The test result is stored (html & json files) stored in separate directory Azure Databricks mount location. The customer wants to attach this /view the result from Azure devops cd pipeline run itself
Hope this clarify.

Hi @hdamczy -In a nutshell,we want to publish the test result in azure devops board instead of hosting separate static website.

Hello @mrajib74 , sorry for the late response. Easter holidays. I’m afraid that I’m not familiar with using azure devops boards and can not help there unfortunately. Would myself be interested in that in the future. :slight_smile:

I think data docs are a red herring here - they are a nice human-readable representation of validation results, but won’t help Azure Devops to know whether or not to proceed.

I am working with an Azure Data Factory pipeline rather than Azure Devops, but have a similar requirement to continue or abort the pipeline depending on the validation results. My solution is to write the results to a Postgres database from my Databricks notebook, then interrogate that database from a subsequent task in my pipeline.

1 Like