# 208dongpha/ecommerce architecture

> Architecture diagram of the GitHub repository 208dongpha/ecommerce, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/208dongpha/ecommerce
- Repository: https://github.com/208dongpha/ecommerce

## Overview

This is a local e-commerce lakehouse analytics pipeline. Its main workflow generates synthetic events, streams them through Bronze and Silver Delta layers, aggregates daily/category metrics into Gold, then serves those metrics through Trino and Superset. Airflow schedules event generation and requests Gold refreshes. The graph preserves README-documented integrations and capabilities; source excerpts confirm producer, streaming, transformation, and batch behavior, while the runner, Trino, and Superset links rely on README documentation.

Main entry points: [`airflow/dags/ecommerce_pipeline.py`](https://github.com/208dongpha/ecommerce/blob/main/airflow/dags/ecommerce_pipeline.py) (Airflow DAG), [`spark/apps/streaming_job.py`](https://github.com/208dongpha/ecommerce/blob/main/spark/apps/streaming_job.py) (Bronze Stream), [`spark/apps/silver_job.py`](https://github.com/208dongpha/ecommerce/blob/main/spark/apps/silver_job.py) (Silver Stream), [`spark/apps/batch_job.py`](https://github.com/208dongpha/ecommerce/blob/main/spark/apps/batch_job.py) (Gold Batch Job)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_airflow["Airflow DAG"]
  node_triggerfiles["Gold Trigger Files"]
end

subgraph group_ingestion["Event Ingestion"]
  node_producer["Event Producer<br/>[producer.py]"]
  node_kafka["Kafka"]
end

subgraph group_lakehouse["Lakehouse Processing"]
  node_bronze["Bronze Stream<br/>[streaming_job.py]"]
  node_bronze_delta[("Bronze Delta")]
  node_silver["Silver Stream<br/>[silver_job.py]"]
  node_silver_delta[("Silver Delta")]
  node_gold_runner["Gold Runner<br/>[gold_runner.sh]"]
  node_batch["Gold Batch Job<br/>[batch_job.py]"]
  node_gold_delta[("Gold Delta")]
end

subgraph group_serving["Analytics Serving"]
  node_trino["Trino SQL<br/>[delta.properties]"]
  node_superset["Superset BI"]
  node_dashboard["Sales Dashboard"]
end

node_operator(("Pipeline Operator"))

node_operator -->|"triggers DAG"| node_airflow
node_airflow -->|"runs"| node_producer
node_producer -->|"publishes events"| node_kafka
node_airflow -->|"allows processing"| node_bronze
node_airflow -->|"waits for processing"| node_silver
node_bronze -->|"reads events"| node_kafka
node_bronze -->|"writes records"| node_bronze_delta
node_silver -->|"reads records"| node_bronze_delta
node_silver -->|"writes cleaned events"| node_silver_delta
node_airflow -->|"writes request"| node_triggerfiles
node_triggerfiles -->|"signals batch run"| node_gold_runner
node_gold_runner -->|"runs job"| node_batch
node_batch -->|"reads events"| node_silver_delta
node_batch -->|"writes metrics"| node_gold_delta
node_trino -->|"queries table"| node_gold_delta
node_superset -->|"queries metrics"| node_trino
node_superset -->|"visualizes analytics"| node_dashboard

click node_airflow "https://github.com/208dongpha/ecommerce/blob/main/airflow/dags/ecommerce_pipeline.py"
click node_triggerfiles "https://github.com/208dongpha/ecommerce/tree/main/airflow/triggers"
click node_producer "https://github.com/208dongpha/ecommerce/blob/main/producer/producer.py"
click node_bronze "https://github.com/208dongpha/ecommerce/blob/main/spark/apps/streaming_job.py"
click node_silver "https://github.com/208dongpha/ecommerce/blob/main/spark/apps/silver_job.py"
click node_gold_runner "https://github.com/208dongpha/ecommerce/blob/main/spark/apps/gold_runner.sh"
click node_batch "https://github.com/208dongpha/ecommerce/blob/main/spark/apps/batch_job.py"
click node_trino "https://github.com/208dongpha/ecommerce/blob/main/trino/catalog/delta.properties"
click node_superset "https://github.com/208dongpha/ecommerce/tree/main/superset"
click node_dashboard "https://github.com/208dongpha/ecommerce/blob/main/dashboards/Ecommerce%20Dashboardv1.html"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_airflow,node_triggerfiles toneBlue
class node_producer,node_kafka toneAmber
class node_bronze,node_bronze_delta,node_silver,node_silver_delta,node_gold_runner,node_batch,node_gold_delta toneMint
class node_trino,node_superset,node_dashboard toneRose
class node_operator toneIndigo
```

## Components

### Orchestration

- [Airflow DAG](https://github.com/208dongpha/ecommerce/blob/main/airflow/dags/ecommerce_pipeline.py): `airflow/dags/ecommerce_pipeline.py`
- [Gold Trigger Files](https://github.com/208dongpha/ecommerce/tree/main/airflow/triggers): `airflow/triggers`

### Event Ingestion

- [Event Producer](https://github.com/208dongpha/ecommerce/blob/main/producer/producer.py): `producer/producer.py`
- **Kafka**

### Lakehouse Processing

- [Bronze Stream](https://github.com/208dongpha/ecommerce/blob/main/spark/apps/streaming_job.py): `spark/apps/streaming_job.py`
- **Bronze Delta**
- [Silver Stream](https://github.com/208dongpha/ecommerce/blob/main/spark/apps/silver_job.py): `spark/apps/silver_job.py`
- **Silver Delta**
- [Gold Runner](https://github.com/208dongpha/ecommerce/blob/main/spark/apps/gold_runner.sh): `spark/apps/gold_runner.sh`
- [Gold Batch Job](https://github.com/208dongpha/ecommerce/blob/main/spark/apps/batch_job.py): `spark/apps/batch_job.py`
- **Gold Delta**

### Analytics Serving

- [Trino SQL](https://github.com/208dongpha/ecommerce/blob/main/trino/catalog/delta.properties): `trino/catalog/delta.properties`
- [Superset BI](https://github.com/208dongpha/ecommerce/tree/main/superset): `superset`
- [Sales Dashboard](https://github.com/208dongpha/ecommerce/blob/main/dashboards/Ecommerce%20Dashboardv1.html): `dashboards/Ecommerce Dashboardv1.html`

### Other components

- **Pipeline Operator**

## Connections

- Pipeline Operator → Airflow DAG: triggers DAG (evidence: `README.md`)
- Airflow DAG → Event Producer: runs (evidence: `airflow/dags/ecommerce_pipeline.py`)
- Event Producer → Kafka: publishes events (evidence: `producer/producer.py`)
- Airflow DAG → Bronze Stream: allows processing (evidence: `README.md`)
- Airflow DAG → Silver Stream: waits for processing (evidence: `README.md`)
- Bronze Stream → Kafka: reads events (evidence: `spark/apps/streaming_job.py`)
- Bronze Stream → Bronze Delta: writes records (evidence: `spark/apps/streaming_job.py`)
- Silver Stream → Bronze Delta: reads records (evidence: `spark/apps/silver_job.py`)
- Silver Stream → Silver Delta: writes cleaned events (evidence: `spark/apps/silver_job.py`)
- Airflow DAG → Gold Trigger Files: writes request (evidence: `airflow/dags/ecommerce_pipeline.py`)
- Gold Trigger Files → Gold Runner: signals batch run (evidence: `README.md`)
- Gold Runner → Gold Batch Job: runs job (evidence: `README.md`)
- Gold Batch Job → Silver Delta: reads events (evidence: `spark/apps/batch_job.py`)
- Gold Batch Job → Gold Delta: writes metrics (evidence: `spark/apps/batch_job.py`)
- Trino SQL → Gold Delta: queries table (evidence: `README.md`)
- Superset BI → Trino SQL: queries metrics (evidence: `README.md`)
- Superset BI → Sales Dashboard: visualizes analytics (evidence: `README.md`)