# karcot1/dataform_deployment_sample architecture

> Architecture diagram of the GitHub repository karcot1/dataform_deployment_sample, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/karcot1/dataform_deployment_sample
- Repository: https://github.com/karcot1/dataform_deployment_sample

## Overview

##### Main components
- Source repository
– GitHub (or other VCS) hosting: contains SQLX code in `definitions/`, `dataform.json`, Cloud Build config (`cloudbuild_*.yaml`), helper scripts (`run.sh`, `run_tests.sh`), and Node manifest (`package.json`).
- CI/CD / Orchestration
– Cloud Build triggers:
- account (manual or on‑commit) uses `cloudbuild_account.yaml`
- customer (Pub/Sub trigger) uses `cloudbuild_customer.yaml`
- sales (Pub/Sub trigger) uses `cloudbuild_sales.yaml`
– Pub/Sub topic “dataform-deployments” for chaining triggers.
- Dataform CLI
– Runs inside each Cloud Build step, executing `dataform run` with the subject‑area tag.
- BigQuery
– Executes SQL jobs generated by Dataform: table creation, views, tests, assertions.
- Pub/Sub
– Receives “success” message from each Cloud Build (with attribute `subjectArea`), which triggers the next build via Cloud Build subscription.

##### Relationships & data flow
1. Developer push → GitHub → triggers (or manual) Cloud Build “account”.
2. Cloud Build “account”:
a. (Optional) `run_tests.sh` for SQLX unit tests
b. `dataform run --tags account` → BigQuery jobs
c. `dataform run --tags account_assertion` → Data quality checks
d. Publish to Pub/Sub “dataform‑deployments” with attribute subjectArea=ACCOUNT
3. Pub/Sub “dataform‑deployments” → Cloud Build “customer” trigger (filter on subjectArea=ACCOUNT).
4. Cloud Build “customer”:
a. `dataform run --tags customer` → BigQuery jobs
b. Publish to Pub/Sub with subjectArea=CUSTOMER
5. Pub/Sub → triggers Cloud Build “sales” (filter on CUSTOMER).
6. Cloud Build “sales”:
a. `dataform run --tags sales` → BigQuery jobs
b. Publish to Pub/Sub with subjectArea=SALES (optional).

##### Architectural patterns & principles
- Event‑driven, decoupled orchestration using Pub/Sub as message bus.
- Tag‑based isolation: Dataform tags prevent cross‑subject interference.
- CI/CD best practices: separate triggers, manual gating, tests/assertions before production.
- Sequence orchestration: explicit ordering of subject areas.

##### Relevant technologies & config
– Dataform CLI (package.json, dataform.json)
- `definitions/` directory feeding into Dataform CLI in each pipeline.

##### Detail but avoid over‑fragmentation
- Three subject areas separately.
- Unit‑test and assertion steps inside the “account” node.
- Config files (`cloudbuild_*.yaml`, `dataform.json`, `package.json`) as part of the repo node.

– Code in GitHub → Cloud Build pipelines
– Dataform CLI orchestrating BigQuery jobs per subject area
– Event‑driven chaining via Pub/Sub
– CI/CD best practices (tests, assertions, tag isolation)
– GCP services in play (Cloud Build, Pub/Sub, BigQuery)

## Diagram (Mermaid)

```mermaid
flowchart TB
  %% Version Control Layer
  subgraph "Version Control"
    direction TB
    Repo["GitHub Repository"]:::repo
    README["README.md"]:::repo
    GITIGNORE[".gitignore"]:::repo
    DFJSON["dataform.json"]:::repo
    PKG["package.json"]:::repo
    PKGLock["package-lock.json"]:::repo
  end

  %% CI/CD & Orchestration Layer
  subgraph "CI/CD & Orchestration"
    direction TB
    CB_ACCOUNT["Cloud Build: account\n(cloudbuild_account.yaml)"]:::compute
    CB_CUSTOMER["Cloud Build: customer\n(cloudbuild_customer.yaml)"]:::compute
    CB_SALES["Cloud Build: sales\n(cloudbuild_sales.yaml)"]:::compute
    RUNTEST["run_tests.sh"]:::repo
    RUNSCRIPT["run.sh"]:::repo
    PUBSUB["Pub/Sub topic\ndataform-deployments"]:::messaging
  end

  %% Data Processing Layer
  subgraph "Data Processing"
    direction TB
    CLI["Dataform CLI\n(definitions/)"]:::cli
  end

  %% Data Warehouse Layer
  subgraph "Data Warehouse"
    direction TB
    BQ["BigQuery"]:::storage
  end

  %% Connections
  Repo -->|"push / trigger"| CB_ACCOUNT
  CB_ACCOUNT -->|"invokes"| RUNTEST
  CB_ACCOUNT -->|"invokes"| RUNSCRIPT
  CB_ACCOUNT -->|"runs Dataform (account tags)"| CLI
  CLI -->|"executes SQL jobs"| BQ
  CB_ACCOUNT -->|"runs assertions"| CLI
  CLI -->|"executes assertions"| BQ
  CB_ACCOUNT -->|"publishes to"| PUBSUB

  PUBSUB -->|"triggers (subjectArea=ACCOUNT)"| CB_CUSTOMER
  CB_CUSTOMER -->|"runs Dataform (customer tags)"| CLI
  CLI -->|"executes SQL jobs"| BQ
  CB_CUSTOMER -->|"publishes to"| PUBSUB

  PUBSUB -->|"triggers (subjectArea=CUSTOMER)"| CB_SALES
  CB_SALES -->|"runs Dataform (sales tags)"| CLI
  CLI -->|"executes SQL jobs"| BQ
  CB_SALES -->|"publishes to"| PUBSUB

  %% Click Events
  click README "https://github.com/karcot1/dataform_deployment_sample/blob/main/README.md"
  click GITIGNORE "https://github.com/karcot1/dataform_deployment_sample/blob/main/.gitignore"
  click DFJSON "https://github.com/karcot1/dataform_deployment_sample/blob/main/dataform.json"
  click PKG "https://github.com/karcot1/dataform_deployment_sample/blob/main/package.json"
  click PKGLock "https://github.com/karcot1/dataform_deployment_sample/blob/main/package-lock.json"
  click CLI "https://github.com/karcot1/dataform_deployment_sample/tree/main/definitions/"
  click CB_ACCOUNT "https://github.com/karcot1/dataform_deployment_sample/blob/main/cloudbuild_account.yaml"
  click CB_CUSTOMER "https://github.com/karcot1/dataform_deployment_sample/blob/main/cloudbuild_customer.yaml"
  click CB_SALES "https://github.com/karcot1/dataform_deployment_sample/blob/main/cloudbuild_sales.yaml"
  click RUNTEST "https://github.com/karcot1/dataform_deployment_sample/blob/main/run_tests.sh"
  click RUNSCRIPT "https://github.com/karcot1/dataform_deployment_sample/blob/main/run.sh"

  %% Styles
  classDef repo fill:#FFE4B5,stroke:#DAA520,color:#000;
  classDef compute fill:#ADD8E6,stroke:#1E90FF,color:#000;
  classDef messaging fill:#FFD700,stroke:#DAA520,color:#000;
  classDef storage fill:#90EE90,stroke:#008000,color:#000;
  classDef cli fill:#F0E68C,stroke:#BDB76B,color:#000;
```
