How karcot1/dataform_deployment_sample works
Main components
• Source repository
– GitHub (or other VCS) hosting: contains SQLX code in definitions/, dataform.json, Cloud Build config (cloudbuild_*.yaml), helper scripts (run.sh, run_tests.sh), and Node manifest (package.json).
• CI/CD / Orchestration
– Cloud Build triggers:
• account (manual or on‑commit) uses cloudbuild_account.yaml
• customer (Pub/Sub trigger) uses cloudbuild_customer.yaml
• sales (Pub/Sub trigger) uses cloudbuild_sales.yaml
– Pub/Sub topic “dataform-deployments” for chaining triggers.
• Dataform CLI
– Runs inside each Cloud Build step, executing dataform run with the subject‑area tag.
• BigQuery
– Executes SQL jobs generated by Dataform: table creation, views, tests, assertions.
• Pub/Sub
– Receives “success” message from each Cloud Build (with attribute subjectArea), which triggers the next build via Cloud Build subscription.
Relationships & data flow
1. Developer push → GitHub → triggers (or manual) Cloud Build “account”.
2. Cloud Build “account”:
a. (Optional) run_tests.sh for SQLX unit tests
b. dataform run --tags account → BigQuery jobs
c. dataform run --tags account_assertion → Data quality checks
d. Publish to Pub/Sub “dataform‑deployments” with attribute subjectArea=ACCOUNT
3. Pub/Sub “dataform‑deployments” → Cloud Build “customer” trigger (filter on subjectArea=ACCOUNT).
4. Cloud Build “customer”:
a. dataform run --tags customer → BigQuery jobs
b. Publish to Pub/Sub with subjectArea=CUSTOMER
5. Pub/Sub → triggers Cloud Build “sales” (filter on CUSTOMER).
6. Cloud Build “sales”:
a. dataform run --tags sales → BigQuery jobs
b. Publish to Pub/Sub with subjectArea=SALES (optional).
Architectural patterns & principles
• Event‑driven, decoupled orchestration using Pub/Sub as message bus.
• Tag‑based isolation: Dataform tags prevent cross‑subject interference.
• CI/CD best practices: separate triggers, manual gating, tests/assertions before production.
• Sequence orchestration: explicit ordering of subject areas.
Relevant technologies & config
– Dataform CLI (package.json, dataform.json)
• definitions/ directory feeding into Dataform CLI in each pipeline.
Detail but avoid over‑fragmentation
• Three subject areas separately.
• Unit‑test and assertion steps inside the “account” node.
• Config files (cloudbuild_*.yaml, dataform.json, package.json) as part of the repo node.
– Code in GitHub → Cloud Build pipelines
– Dataform CLI orchestrating BigQuery jobs per subject area
– Event‑driven chaining via Pub/Sub
– CI/CD best practices (tests, assertions, tag isolation)
– GCP services in play (Cloud Build, Pub/Sub, BigQuery)