# googlecloudplatform/marketing-analytics-jumpstart-dataform architecture

> Architecture diagram of the GitHub repository googlecloudplatform/marketing-analytics-jumpstart-dataform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/googlecloudplatform/marketing-analytics-jumpstart-dataform
- Repository: https://github.com/googlecloudplatform/marketing-analytics-jumpstart-dataform

## Overview

##### Project Type and Purpose
- This is a Dataform‐based ELT/analytics pipeline project for BigQuery.
- It ingests raw marketing data from Google Analytics 4 (GA4) exports and Google Ads exports, applies transformations (SQLX scripts), and surfaces standardized “product” views for dashboards, analytics, ML training, and GenAI use cases.
- The codebase is structured around two domains (“ga4” and “ads”), each with a “base” layer (staging/processing) and a “product” layer (stable data interfaces).

##### Top‑Level Components
a. External Data Sources
- GA4 export tables (BigQuery)
- Ads export tables (BigQuery via Google Ads Data Transfer Service)
b. Orchestration & Transformation
- Dataform project (dataform.json + definitions/*.sqlx + includes/*.js)
- Custom SQLX definitions per domain (base, product, backfill, views, incremental updates)
- Helper JS modules for shared logic (includes/functions.js, constants.js, etc.)
c. BigQuery Datasets
- Source datasets (raw GA4 & Ads)
- Base datasets: marketing_ga4_base_<env>, marketing_ads_base_<env>
- Product datasets: marketing_ga4_v1_<env>, marketing_ads_v1_<env>
d. Consumers
- BI Dashboards (Looker, Data Studio, etc.)
- Ad hoc analytics & SQL queries
- Machine Learning training pipelines
- GenAI applications

##### Key Architectural Patterns
- ELT Pipeline: Raw data loaded first; transformations pushed down into BigQuery via SQLX.
- Domain‑Driven Layers: Separate “base” (cleaning, shaping) vs. “product” (stable views).
- Environment Promotion: dev / stage / prod suffix on dataset names controlled via dataform.json variables.
- Modular Definitions: SQLX scripts organized by domain and function (source, base, product, backfill, update).
- Reusability & Extensibility: JS helper modules (includes/) and tagging (“ga4”, “ads”) for targeted compilation.

##### Detail Level
- Individual BigQuery dataset names (marketing_ga4_base_dev, marketing_ga4_v1_prod, etc.)
- Two domains as parallel lanes

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Source Layer
    subgraph "Source Layer"
        GA4_src_cloud("GA4 Export Tables"):::external
        Ads_src_cloud("Ads Export Tables"):::external
    end

    %% Transformation Layer
    subgraph "Transformation Layer"
        subgraph "Dataform Project"
            DF_config["dataform.json"]:::code
            DF_defs["definitions/"]:::code
            DF_includes["includes/"]:::code
            DF_sync["mde_repo_sync.sh"]:::code
            subgraph "SQLX Definitions"
                GA4_src["GA4 Source SQLX"]:::ga4
                GA4_base_defs["GA4 Base SQLX"]:::ga4
                GA4_backfill["GA4 Backfill SQLX"]:::ga4
                GA4_prod_defs["GA4 Product v1 SQLX"]:::ga4
                Ads_base_defs["Ads Base SQLX"]:::ads
                Ads_prod_defs["Ads Product v1 SQLX"]:::ads
                Ads_update["Ads Incremental Update SQLX"]:::ads
                Ads_views["Ads View Definitions"]:::ads
            end
        end
    end

    %% Storage Layer
    subgraph "Storage Layer"
        subgraph "GA4 Pipeline"
            G4_base["marketing_ga4_base_<env>"]:::data
            G4_prod["marketing_ga4_v1_<env>"]:::data
        end
        subgraph "Ads Pipeline"
            Ads_base["marketing_ads_base_<env>"]:::data
            Ads_prod["marketing_ads_v1_<env>"]:::data
        end
    end

    %% Consumption Layer
    subgraph "Consumption Layer"
        Dashboards["BI Dashboards"]:::consumer
        Analytics["Ad hoc Analytics"]:::consumer
        ML["ML Training Pipelines"]:::consumer
        GenAI["GenAI Applications"]:::consumer
    end

    %% Connections
    GA4_src_cloud -->|ingest| GA4_src
    Ads_src_cloud -->|ingest| Ads_base_defs

    GA4_src -->|transform| GA4_base_defs
    Ads_base_defs -->|transform| Ads_base_defs

    GA4_base_defs -->|load| G4_base
    Ads_base_defs -->|load| Ads_base

    G4_base -->|view/create| GA4_prod_defs
    Ads_base -->|view/create| Ads_prod_defs

    GA4_prod_defs -->|load| G4_prod
    Ads_prod_defs -->|load| Ads_prod

    G4_prod -->|consume| Dashboards
    G4_prod -->|consume| Analytics
    G4_prod -->|consume| ML
    G4_prod -->|consume| GenAI

    Ads_prod -->|consume| Dashboards
    Ads_prod -->|consume| Analytics
    Ads_prod -->|consume| ML
    Ads_prod -->|consume| GenAI

    %% Click Events
    click DF_config "https://github.com/googlecloudplatform/marketing-analytics-jumpstart-dataform/blob/main/dataform.json"
    click DF_defs "https://github.com/googlecloudplatform/marketing-analytics-jumpstart-dataform/tree/main/definitions/"
    click DF_includes "https://github.com/googlecloudplatform/marketing-analytics-jumpstart-dataform/tree/main/includes/"
    click DF_sync "https://github.com/googlecloudplatform/marketing-analytics-jumpstart-dataform/blob/main/mde_repo_sync.sh"
    click GA4_src "https://github.com/googlecloudplatform/marketing-analytics-jumpstart-dataform/tree/main/definitions/digital_analytics_domain/source/"
    click GA4_base_defs "https://github.com/googlecloudplatform/marketing-analytics-jumpstart-dataform/tree/main/definitions/digital_analytics_domain/base/"
    click GA4_backfill "https://github.com/googlecloudplatform/marketing-analytics-jumpstart-dataform/tree/main/definitions/digital_analytics_domain/backfill/"
    click GA4_prod_defs "https://github.com/googlecloudplatform/marketing-analytics-jumpstart-dataform/tree/main/definitions/digital_analytics_domain/product_v1/"
    click Ads_base_defs "https://github.com/googlecloudplatform/marketing-analytics-jumpstart-dataform/tree/main/definitions/ads_domain/def/"
    click Ads_prod_defs "https://github.com/googlecloudplatform/marketing-analytics-jumpstart-dataform/tree/main/definitions/ads_domain/google_ads/product_v1/"
    click Ads_update "https://github.com/googlecloudplatform/marketing-analytics-jumpstart-dataform/tree/main/definitions/ads_domain/google_ads/update/"
    click Ads_views "https://github.com/googlecloudplatform/marketing-analytics-jumpstart-dataform/tree/main/definitions/ads_domain/google_ads/views/"

    %% Styles
    classDef ga4 fill:#D6EAF8,stroke:#21618C,stroke-width:1px
    classDef ads fill:#FDEBD0,stroke:#B9770E,stroke-width:1px
    classDef code fill:#F5B7B1,stroke:#922B21,stroke-width:1px
    classDef data fill:#ABEBC6,stroke:#1D8348,stroke-width:1px
    classDef external fill:#E8EAF6,stroke:#5D6D7E,stroke-width:1px
    classDef service fill:#D5F5E3,stroke:#117A65,stroke-width:1px
    classDef consumer fill:#FCF3CF,stroke:#B7950B,stroke-width:1px
```
