How googlecloudplatform/marketing-analytics-jumpstart-dataform works
Project Type and Purpose
• This is a Dataform‐based ELT/analytics pipeline project for BigQuery.
• It ingests raw marketing data from Google Analytics 4 (GA4) exports and Google Ads exports, applies transformations (SQLX scripts), and surfaces standardized “product” views for dashboards, analytics, ML training, and GenAI use cases.
• The codebase is structured around two domains (“ga4” and “ads”), each with a “base” layer (staging/processing) and a “product” layer (stable data interfaces).
Top‑Level Components
a. External Data Sources
• GA4 export tables (BigQuery)
• Ads export tables (BigQuery via Google Ads Data Transfer Service)
b. Orchestration & Transformation
• Dataform project (dataform.json + definitions/*.sqlx + includes/*.js)
• Custom SQLX definitions per domain (base, product, backfill, views, incremental updates)
• Helper JS modules for shared logic (includes/functions.js, constants.js, etc.)
c. BigQuery Datasets
• Source datasets (raw GA4 & Ads)
• Base datasets: marketing_ga4_base_<env>, marketing_ads_base_<env>
• Product datasets: marketing_ga4_v1_<env>, marketing_ads_v1_<env>
d. Consumers
• BI Dashboards (Looker, Data Studio, etc.)
• Ad hoc analytics & SQL queries
• Machine Learning training pipelines
• GenAI applications
Key Architectural Patterns
• ELT Pipeline: Raw data loaded first; transformations pushed down into BigQuery via SQLX.
• Domain‑Driven Layers: Separate “base” (cleaning, shaping) vs. “product” (stable views).
• Environment Promotion: dev / stage / prod suffix on dataset names controlled via dataform.json variables.
• Modular Definitions: SQLX scripts organized by domain and function (source, base, product, backfill, update).
• Reusability & Extensibility: JS helper modules (includes/) and tagging (“ga4”, “ads”) for targeted compilation.
Detail Level
• Individual BigQuery dataset names (marketing_ga4_base_dev, marketing_ga4_v1_prod, etc.)
• Two domains as parallel lanes