GX in GCP

GX Core Support
1

I want to upload my gx folder in a GCP bucket. I think it’s the best practice when you publish your expectation in a cloud run. I don’t know how to push my GX folder with vlaidation and expectation and data docs in a bucket.

2

Hi @xelov — this is a common pattern, and the good news is GX doesn’t need anything special here. Treat GCP as the source of truth and let the standard GCP tooling move files in and out; GX itself doesn’t need to know it’s talking to a bucket.

The general flow on each Cloud Run invocation:

  1. Pull your gx/ folder down at the start of the run using the standard Google Cloud Storage client library (or the gsutil/gcloud storage CLI if you’re scripting it). This gives you your expectations, checkpoints, and any existing Data Docs locally in the container’s filesystem.
  2. Do the normal GX workflow against that local folder:
    • Get your DataContext
    • Get your Checkpoint
    • Run validation
  3. Push the whole gx/ folder back up to the bucket afterward, again using the GCS client library. This syncs your updated validation results, run history, and rebuilt Data Docs back to your source of truth.

A couple of things worth knowing about before you build this:

  • Cloud Run containers are ephemeral — nothing persists between invocations unless you explicitly pull it in and push it back out, which is exactly the round-trip above.
  • GCS has a native “sync” concept (gsutil rsync / gcloud storage rsync) that’s usually a better fit for “mirror this whole local folder to/from a bucket” than downloading/uploading files one at a time.
  • If multiple runs could write back concurrently, look at GCS object versioning or generation preconditions so you don’t clobber another run’s results.

None of this is GX-specific configuration — it’s just “hydrate local state from your bucket, run GX normally, persist local state back to the bucket,” using whatever GCP tooling you’re already comfortable with.

3

Thanks for the advice, I’ll follow it. I have another point related to this. I’d like to share a single GX setup across three tests: I have one Cloud Run to test the bronze data, another for silver, and a last one for gold. Is it best practice to test data with GX across three separate Cloud Functions? And I’m not sure how it would work with a shared GX stored in a bucket.