# velmurugan27s/ecommerce-elt-platform architecture

> Architecture diagram of the GitHub repository velmurugan27s/ecommerce-elt-platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/velmurugan27s/ecommerce-elt-platform
- Repository: https://github.com/velmurugan27s/ecommerce-elt-platform

## Overview

This repository is an e-commerce ELT platform: a scheduled Airflow DAG coordinates CSV ingestion into MinIO Bronze, Spark processing into Silver, PostgreSQL staging loads, and dbt transformations into analytics models. Its modeled outputs include customer, product, seller, and date dimensions plus order-item, payment, and review facts. Ingestion analysis scripts are a separate supporting capability. Evidence is strongest for orchestration and the sampled ingestion/Spark/load code; individual dbt model dependencies and unsampled analytics consumers are not shown.

Main entry points: [`dags/ecommerce_elt_pipeline.py`](https://github.com/velmurugan27s/ecommerce-elt-platform/blob/main/dags/ecommerce_elt_pipeline.py) (ELT Orchestrator), [`ingestion/upload_to_bronze.py`](https://github.com/velmurugan27s/ecommerce-elt-platform/blob/main/ingestion/upload_to_bronze.py) (Bronze Ingestion), [`spark/jobs/process_bronze_to_silver.py`](https://github.com/velmurugan27s/ecommerce-elt-platform/blob/main/spark/jobs/process_bronze_to_silver.py) (Silver Processing), [`ingestion/load_silver_to_postgres.py`](https://github.com/velmurugan27s/ecommerce-elt-platform/blob/main/ingestion/load_silver_to_postgres.py) (Postgres Loader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_airflow_dag["ELT Orchestrator"]
  node_batch_id["Batch Validation"]
end

subgraph group_data_lake["Lake Processing"]
  node_bronze_ingest["Bronze Ingestion"]
  node_minio_client["MinIO Client<br/>[minio_client.py]"]
  node_minio[("MinIO Object Store")]
  node_spark_silver["Silver Processing"]
end

subgraph group_warehouse["Warehouse Modeling"]
  node_postgres_load["Postgres Loader"]
  node_postgres[("PostgreSQL Warehouse")]
  node_dbt_staging["Staging Models"]
  node_dbt_marts["Analytics Marts"]
  node_dimensions["Dimensions"]
  node_facts["Fact Models"]
end

subgraph group_analysis["Data Analysis"]
  node_ingestion_analysis["Dataset Analysis"]
end

node_source_data["Source CSVs"]

node_airflow_dag -->|"runs ingestion"| node_bronze_ingest
node_source_data -->|"uploads CSVs"| node_bronze_ingest
node_bronze_ingest -->|"uses client"| node_minio_client
node_minio_client -->|"stores objects"| node_minio
node_bronze_ingest -->|"emits batch ID"| node_batch_id
node_batch_id -->|"passes batch"| node_spark_silver
node_airflow_dag -->|"runs processing"| node_spark_silver
node_spark_silver -->|"reads Bronze, writes Silver"| node_minio
node_airflow_dag -->|"runs loader"| node_postgres_load
node_spark_silver -->|"precedes load"| node_postgres_load
node_postgres_load -->|"reads Silver"| node_minio
node_postgres_load -->|"loads staging"| node_postgres
node_airflow_dag -->|"runs dbt"| node_dbt_staging
node_airflow_dag -->|"configures dbt connection"| node_postgres
node_dbt_staging -->|"transforms to marts"| node_dbt_marts
node_dbt_marts -.->|"defines dimensions"| node_dimensions
node_dbt_marts -.->|"defines facts"| node_facts

click node_airflow_dag "https://github.com/velmurugan27s/ecommerce-elt-platform/blob/main/dags/ecommerce_elt_pipeline.py"
click node_batch_id "https://github.com/velmurugan27s/ecommerce-elt-platform/blob/main/dags/ecommerce_elt_pipeline.py"
click node_bronze_ingest "https://github.com/velmurugan27s/ecommerce-elt-platform/blob/main/ingestion/upload_to_bronze.py"
click node_minio_client "https://github.com/velmurugan27s/ecommerce-elt-platform/blob/main/ingestion/minio_client.py"
click node_spark_silver "https://github.com/velmurugan27s/ecommerce-elt-platform/blob/main/spark/jobs/process_bronze_to_silver.py"
click node_postgres_load "https://github.com/velmurugan27s/ecommerce-elt-platform/blob/main/ingestion/load_silver_to_postgres.py"
click node_dbt_staging "https://github.com/velmurugan27s/ecommerce-elt-platform/tree/main/dbt/models/staging"
click node_dbt_marts "https://github.com/velmurugan27s/ecommerce-elt-platform/tree/main/dbt/models/marts"
click node_dimensions "https://github.com/velmurugan27s/ecommerce-elt-platform/tree/main/dbt/models/marts"
click node_facts "https://github.com/velmurugan27s/ecommerce-elt-platform/tree/main/dbt/models/marts"
click node_ingestion_analysis "https://github.com/velmurugan27s/ecommerce-elt-platform/tree/main/ingestion"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_airflow_dag,node_batch_id toneBlue
class node_bronze_ingest,node_minio_client,node_minio,node_spark_silver toneAmber
class node_postgres_load,node_postgres,node_dbt_staging,node_dbt_marts,node_dimensions,node_facts toneMint
class node_ingestion_analysis toneRose
class node_source_data toneIndigo
```

## Components

### Orchestration

- [ELT Orchestrator](https://github.com/velmurugan27s/ecommerce-elt-platform/blob/main/dags/ecommerce_elt_pipeline.py): `dags/ecommerce_elt_pipeline.py`
- [Batch Validation](https://github.com/velmurugan27s/ecommerce-elt-platform/blob/main/dags/ecommerce_elt_pipeline.py): `dags/ecommerce_elt_pipeline.py`

### Lake Processing

- [Bronze Ingestion](https://github.com/velmurugan27s/ecommerce-elt-platform/blob/main/ingestion/upload_to_bronze.py): `ingestion/upload_to_bronze.py`
- [MinIO Client](https://github.com/velmurugan27s/ecommerce-elt-platform/blob/main/ingestion/minio_client.py): `ingestion/minio_client.py`
- **MinIO Object Store**
- [Silver Processing](https://github.com/velmurugan27s/ecommerce-elt-platform/blob/main/spark/jobs/process_bronze_to_silver.py): `spark/jobs/process_bronze_to_silver.py`

### Warehouse Modeling

- [Postgres Loader](https://github.com/velmurugan27s/ecommerce-elt-platform/blob/main/ingestion/load_silver_to_postgres.py): `ingestion/load_silver_to_postgres.py`
- **PostgreSQL Warehouse**
- [Staging Models](https://github.com/velmurugan27s/ecommerce-elt-platform/tree/main/dbt/models/staging): `dbt/models/staging`
- [Analytics Marts](https://github.com/velmurugan27s/ecommerce-elt-platform/tree/main/dbt/models/marts): `dbt/models/marts`
- [Dimensions](https://github.com/velmurugan27s/ecommerce-elt-platform/tree/main/dbt/models/marts): `dbt/models/marts`
- [Fact Models](https://github.com/velmurugan27s/ecommerce-elt-platform/tree/main/dbt/models/marts): `dbt/models/marts`

### Data Analysis

- [Dataset Analysis](https://github.com/velmurugan27s/ecommerce-elt-platform/tree/main/ingestion): `ingestion`

### Other components

- **Source CSVs**

## Connections

- ELT Orchestrator → Bronze Ingestion: runs ingestion (evidence: `dags/ecommerce_elt_pipeline.py`)
- Source CSVs → Bronze Ingestion: uploads CSVs (evidence: `ingestion/upload_to_bronze.py`)
- Bronze Ingestion → MinIO Client: uses client (evidence: `ingestion/upload_to_bronze.py`)
- MinIO Client → MinIO Object Store: stores objects (evidence: `ingestion/minio_client.py`)
- Bronze Ingestion → Batch Validation: emits batch ID (evidence: `dags/ecommerce_elt_pipeline.py`)
- Batch Validation → Silver Processing: passes batch (evidence: `dags/ecommerce_elt_pipeline.py`)
- ELT Orchestrator → Silver Processing: runs processing (evidence: `dags/ecommerce_elt_pipeline.py`)
- Silver Processing → MinIO Object Store: reads Bronze, writes Silver (evidence: `spark/jobs/process_bronze_to_silver.py`)
- ELT Orchestrator → Postgres Loader: runs loader (evidence: `dags/ecommerce_elt_pipeline.py`)
- Silver Processing → Postgres Loader: precedes load
- Postgres Loader → MinIO Object Store: reads Silver (evidence: `ingestion/load_silver_to_postgres.py`)
- Postgres Loader → PostgreSQL Warehouse: loads staging (evidence: `ingestion/load_silver_to_postgres.py`)
- ELT Orchestrator → Staging Models: runs dbt (evidence: `dags/ecommerce_elt_pipeline.py`)
- ELT Orchestrator → PostgreSQL Warehouse: configures dbt connection (evidence: `dags/ecommerce_elt_pipeline.py`)
- Staging Models → Analytics Marts: transforms to marts
- Analytics Marts → Dimensions: defines dimensions
- Analytics Marts → Fact Models: defines facts
