Is there a way to use Service Principal to connect to Azure blob storage instead of access key and SAS token in great expectations?

GX Core (OSS) Support
1

Hi Team,

Is there a way to connect to azure blob storage or ADLS using Service principal instead of access key or SAS token. PFB the code snippet I am using to connect to ADLS using Access key or SAS token.

context = gx.get_context(context_root_dir=contextDirecotry)

#using SAS token
storageAccountURL = f"https://{storageAccountName}.blob.core.windows.net/?{storage_account_sas_token}"
azure_options_sas = {"account_url": storageAccountURL}
datasource_sas = context.sources.add_or_update_spark_abs(name=datasource_name, azure_options=azure_options_sas)

#using storage account access key
storageAccountConnectionString = f"DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net;AccountName={storageAccountName};AccountKey={storage_account_access_key}"
azure_options_account_key = {"conn_str": storageAccountConnectionString}
datasource_account_key = context.sources.add_or_update_spark_abs(name=datasource_name, azure_options=azure_options_account_key)

Is there any other way to connect to blob storage using Service principal authentication (client id, tenant id, client secret)

Thanks,
Dinesh

2

i have a same query, i was working on the proof of value in my project and spent a lot of time setting up everything in databricks. Now i am having trouble in setting up the data docs as i need to pass in the access key and token of s3 bucket where i am hosting the data docs which i can’t as my organization uses i am roles
Note: @rodger67 originally posted this reply in Slack. It might not have transferred perfectly.

3

<@U06TC1ZMX1A> I am able to use access key and token in Azure using great expectations but not service principal.
For AWS S3 bucket you can refer this <https://docs.greatexpectations.io/docs/oss/deployment_patterns/how_to_use_gx_with_aws/how_to_use_gx_with_aws_using_s3_and_spark/|Use Great Expectations with Amazon Web Services using S3 and Spark>.
Note: @Dinesh Baratam originally posted this reply in Slack. It might not have transferred perfectly.

4

Thanks for the reply <@U075BU5FKT4> but my issue is slightly different. I can’t use access key as my organization does not use it for access to the s3 bucket instead use i am roles due to which i cant write the data docs directly from my code as gx expects these keys
Note: @rodger67 originally posted this reply in Slack. It might not have transferred perfectly.