How do I use rotating secrets from AWS Secrets Manager with GE's datasource credentials?

Archive
2

Was able to figure this out with some help.

  1. Configure the Redshift datasource to use environmental variables, as Eugene explained in this helpful post.
datasources:
    datawarehouse:
        class_name: SqlAlchemyDatasource
        data_asset_type:
            class_name: SqlAlchemyDataset
        module_name:
        credentials:
            drivername: postgresql+psycopg2
            host: myRedshiftHost
            port: '5439'
            database: myRedshiftDb
            username: ${GE_REDSHIFT_USERNAME}
            password: ${GE_REDSHIFT_PASSWORD}
  1. Before invoking GE, set environmental variables using a separate python script. See this StackOverflow post for an explanation of how to do in boto3.
    cluster_creds = boto3.client('redshift').get_cluster_credentials(DbUser=RedshiftUser,
                                               DbName=RedshiftDb,
                                          ClusterIdentifier=RedshiftClusterId,
                                               AutoCreate=False)
os.environ['GE_REDSHIFT_USERNAME'] = cluster_creds['DbUser']
os.environ['GE_REDSHIFT_PASSWORD'] = cluster_creds['DbPassword']
2 Likes