# rehaman24/retail-etl-pipeline-airflow-gcp architecture

> Architecture diagram of the GitHub repository rehaman24/retail-etl-pipeline-airflow-gcp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rehaman24/retail-etl-pipeline-airflow-gcp
- Repository: https://github.com/rehaman24/retail-etl-pipeline-airflow-gcp

## Overview

##### Project Type & Purpose
- This is a cloud-native ETL pipeline (full-stack data engineering solution) that ingests merchant and sales JSON files from Google Cloud Storage (GCS), orchestrates multi-stage tasks with Apache Airflow, and loads, transforms, and merges data into a star-schema in BigQuery. Its goal is to deliver analytics-ready retail data for BI and reporting.

##### File Structure
- Top-level directories:
- data/merchants and data/sales contain sample JSON files (staging inputs).
- screenshots/ capture Airflow, BigQuery, and query results (documentation).
- Key external components (implied by README):
- dags/airflow_walmart_data_bigquery_dag.py (Airflow DAG definition).
- requirements.txt and connection/config files for GCP.

##### Extract Architecture Insights from README
- Core stages:
a. Dataset & Table creation via BigQuery operators
b. Parallel load tasks (GCSToBigQueryOperator)
c. Merge/UPSERT via BigQueryInsertJobOperator
- Dimensional model: star schema with a merchants dimension and a fact table (walmart_sales_tgt) that denormalizes merchant attributes.
- TaskGroup used for parallelism; idempotent design to allow safe re-runs.

a. Main Components:
- GCS Bucket (raw JSON storage)
- Airflow Orchestration (DAG hosted on Cloud Composer or self-managed Airflow)
- BigQuery Data Warehouse (walmart_dwh dataset with staging, dimension, fact tables)
- BI/Visualization Layer (Looker/Tableau/Power BI)—optional, for end-users
b. Relationships & Data Flow:
1. JSON files land in GCS under /merchants and /sales
2. Airflow DAG triggers at midnight UTC
3. Dataset creation → runtime table creation → parallel data load → MERGE task
4. Final tables serve analytics queries or feed into BI tools
c. Architectural Patterns & Principles:
- Star Schema (dimension + fact) for analytical performance
- Staging Layer to decouple ingestion from transformation
- MERGE-based UPSERT for idempotent, atomic updates
- TaskGroup for parallel execution and reduced DAG runtime
- Infrastructure as code: operators define dataset/table schema programmatically
d. Technologies & Frameworks:
- Orchestration: Apache Airflow 2.7+ (TaskGroup, BigQuery/GCS operators)
- Storage: Google Cloud Storage (NDJSON files)
- Warehouse: BigQuery (Standard SQL, MERGE)
- Language: Python 3.8+ for DAG, SQL for transformations

##### Detailed Elements to Capture
- Exact DAG schedule & DAG ID (walmart_sales_etl_gcs)
- Task dependencies graph: create_dataset → [create_tables] → load_data(TaskGroup) → merge_walmart_sales
- GCS paths: gs://…/merchants/*.json and gs://…/sales/*.json
- BigQuery dataset “Walmart_Dwh” with its three tables
- BI layer connection (optional): direct querying of walmart_sales_tgt

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Raw Storage (GCS)"
        GCS1["merchants JSON (gs://.../merchants/*.json)"]:::storage
        GCS2["sales JSON (gs://.../sales/*.json)"]:::storage
    end
    Docs1["Sample Data Documentation"]:::external
    click Docs1 "https://github.com/rehaman24/retail-etl-pipeline-airflow-gcp/blob/main/data/README.md"
    Docs1 -.-> GCS1
    Docs1 -.-> GCS2

    GCS1 --> Airflow
    GCS2 --> Airflow

    Airflow["Airflow DAG (walmart_sales_etl_gcs)"]:::orchestration
    click Airflow "https://github.com/rehaman24/retail-etl-pipeline-airflow-gcp/blob/main/README.md"
    Docs2["Airflow UI Screenshots"]:::external
    click Docs2 "https://github.com/rehaman24/retail-etl-pipeline-airflow-gcp/blob/main/screenshots/airflow/README.md"
    Docs2 -.-> Airflow

    Airflow --> CreateDS
    CreateDS["create_dataset"]:::compute
    CreateDS --> CreateTables
    CreateTables["create_tables"]:::compute

    CreateTables --> LoadTG
    subgraph "Load Data TaskGroup"
        direction TB
        LD1["Load merchants staging"]:::compute
        LD2["Load sales staging"]:::compute
        LoadTG["TaskGroup: Load Data"]:::compute
        LoadTG --> LD1
        LoadTG --> LD2
    end

    LoadTG --> Merge
    Merge["merge_walmart_sales (MERGE)"]:::compute
    Merge --> BigQuery

    subgraph "BigQuery (Walmart_Dwh)"
        direction TB
        ST1["walmart_merchants_stage"]:::bqtable
        ST2["walmart_sales_stage"]:::bqtable
        MD["merchant_dim"]:::bqtable
        FT["walmart_sales_tgt"]:::bqtable
    end

    Docs3["BigQuery Schema Screenshots"]:::external
    click Docs3 "https://github.com/rehaman24/retail-etl-pipeline-airflow-gcp/blob/main/screenshots/bigquery/README.md"
    Docs3 -.-> ST1
    Docs3 -.-> ST2
    Docs3 -.-> MD
    Docs3 -.-> FT

    Merge --> FT
    FT --> BI
    Docs4["Query Results Documentation"]:::external
    click Docs4 "https://github.com/rehaman24/retail-etl-pipeline-airflow-gcp/blob/main/screenshots/Query_results/README.md"
    Docs4 -.-> FT

    BI["BI/Visualization Layer"]:::external

    classDef storage fill:#87CEEB,stroke:#333,stroke-width:1px;
    classDef orchestration fill:#98FB98,stroke:#333,stroke-width:1px;
    classDef compute fill:#90EE90,stroke:#333,stroke-width:1px;
    classDef bqtable fill:#ADD8E6,stroke:#333,stroke-width:1px;
    classDef external fill:#FFD700,stroke:#333,stroke-width:1px;
```
