I am using GE to run assertions against a dataset in Redshift!
Our Redshift cluster requires us to use rotating secrets via AWS secrets manager. In this connection method, an IAM user is granted a temporary username and password with access to our Redshift cluster for 900 seconds. This secret is obtained via a method in boto3 (get_cluster_credentials)
My understanding is that GE datasources can currently only be configured to use hardcoded credentials. Is it possible to run GE using temporary credentials obtained at runtime?
Was able to figure this out with some help.
- Configure the Redshift datasource to use environmental variables, as Eugene explained in this helpful post.
- Before invoking GE, set environmental variables using a separate python script. See this StackOverflow post for an explanation of how to do in boto3.
cluster_creds = boto3.client('redshift').get_cluster_credentials(DbUser=RedshiftUser,
os.environ['GE_REDSHIFT_USERNAME'] = cluster_creds['DbUser']
os.environ['GE_REDSHIFT_PASSWORD'] = cluster_creds['DbPassword']