# hsai-rohit/real-time-ecommerce-data-platform architecture

> Architecture diagram of the GitHub repository hsai-rohit/real-time-ecommerce-data-platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hsai-rohit/real-time-ecommerce-data-platform
- Repository: https://github.com/hsai-rohit/real-time-ecommerce-data-platform

## Overview

This is a batch-oriented e-commerce data ingestion and validation platform, with local CSV, SQLite, and mock-API sources plus ADLS Gen2 and Databricks integrations. The principal workflow is to run a source-specific ingestion command, land raw data, validate its contracts and relationships, then optionally upload it to cloud storage. The excerpts show several runner-to-ingester calls and validation behavior; unsampled orchestration and validation runners are represented without asserted call links.

Main entry points: [`data`](https://github.com/hsai-rohit/real-time-ecommerce-data-platform/tree/main/data) (Raw Data Landing), [`ingestion/api_ingestion.py`](https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/ingestion/api_ingestion.py) (API Landing), [`validation/csv_validation.py`](https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/validation/csv_validation.py) (CSV Contract Checks), [`validation/sqlite_validation.py`](https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/validation/sqlite_validation.py) (Orders & Payments Checks)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Source Ingestion"]
  node_csvrunner["CSV Ingestion Runner"]
  node_csvingest["CSV Landing<br/>[csv_ingestion.py]"]
  node_sqlrunner["SQLite Ingestion Runner"]
  node_sqlingest["SQLite Extraction"]
  node_apirunner["API Ingestion Runner"]
  node_apiingest["API Landing<br/>[api_ingestion.py]"]
  node_rawdata[("Raw Data Landing")]
end

subgraph group_validate["Data Validation"]
  node_csvvalidation["CSV Contract Checks<br/>[csv_validation.py]"]
  node_sqlvalidation["Orders &amp; Payments Checks"]
  node_apivalidation["Events API Checks<br/>[api_validation.py]"]
end

subgraph group_cloud["Cloud Storage"]
  node_adlsupload["ADLS Upload<br/>[adls_upload.py]"]
  node_adlsdownload["ADLS Download<br/>[adls_download.py]"]
  node_databricksupload["Databricks Volume Upload"]
end

subgraph group_sources["Local Sources"]
  node_csvsource["Customer &amp; Product CSV"]
  node_sqlitesource[("Orders &amp; Payments DB")]
  node_eventsapi["Mock Events API<br/>[mock_events_api.py]"]
end

node_operator(("Operator"))
node_adls[("Azure ADLS Gen2")]
node_databricks[("Databricks Volume")]

node_operator -.->|"runs"| node_csvrunner
node_operator -.->|"runs"| node_sqlrunner
node_operator -.->|"runs"| node_apirunner
node_csvrunner -->|"calls"| node_csvingest
node_sqlrunner -->|"calls"| node_sqlingest
node_apirunner -->|"calls"| node_apiingest
node_csvingest -->|"writes"| node_rawdata
node_sqlingest -->|"writes"| node_rawdata
node_apiingest -->|"fetches JSON"| node_eventsapi
node_apiingest -->|"writes"| node_rawdata
node_csvvalidation -.->|"checks CSV"| node_rawdata
node_sqlvalidation -.->|"checks CSV"| node_rawdata
node_apivalidation -.->|"checks JSON"| node_rawdata
node_apivalidation -->|"uses result type"| node_csvvalidation
node_sqlvalidation -->|"uses helpers"| node_csvvalidation
node_operator -.->|"runs upload"| node_adlsupload
node_adlsupload -->|"uploads files"| node_adls
node_adls -->|"downloads prefix"| node_adlsdownload
node_operator -.->|"runs download"| node_adlsdownload
node_operator -.->|"runs upload"| node_databricksupload
node_databricksupload -->|"uploads files"| node_databricks

click node_csvsource "https://github.com/hsai-rohit/real-time-ecommerce-data-platform/tree/main/data/source/csv"
click node_sqlitesource "https://github.com/hsai-rohit/real-time-ecommerce-data-platform/tree/main/data/source/sqlite"
click node_eventsapi "https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/scripts/mock_events_api.py"
click node_csvrunner "https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/scripts/run_csv_ingestion.py"
click node_csvingest "https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/ingestion/csv_ingestion.py"
click node_sqlrunner "https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/scripts/run_sqlite_ingestion.py"
click node_sqlingest "https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/ingestion/sqlite_ingestion.py"
click node_apirunner "https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/scripts/run_api_ingestion.py"
click node_apiingest "https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/ingestion/api_ingestion.py"
click node_rawdata "https://github.com/hsai-rohit/real-time-ecommerce-data-platform/tree/main/data"
click node_csvvalidation "https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/validation/csv_validation.py"
click node_sqlvalidation "https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/validation/sqlite_validation.py"
click node_apivalidation "https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/validation/api_validation.py"
click node_adlsupload "https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/ingestion/adls_upload.py"
click node_adlsdownload "https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/ingestion/adls_download.py"
click node_databricksupload "https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/ingestion/databricks_volume_upload.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_csvrunner,node_csvingest,node_sqlrunner,node_sqlingest,node_apirunner,node_apiingest,node_rawdata toneBlue
class node_csvvalidation,node_sqlvalidation,node_apivalidation,node_adls,node_databricks toneAmber
class node_adlsupload,node_adlsdownload,node_databricksupload toneMint
class node_csvsource,node_sqlitesource,node_eventsapi toneRose
class node_operator toneIndigo
```

## Components

### Source Ingestion

- [CSV Ingestion Runner](https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/scripts/run_csv_ingestion.py): `scripts/run_csv_ingestion.py`
- [CSV Landing](https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/ingestion/csv_ingestion.py): `ingestion/csv_ingestion.py`
- [SQLite Ingestion Runner](https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/scripts/run_sqlite_ingestion.py): `scripts/run_sqlite_ingestion.py`
- [SQLite Extraction](https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/ingestion/sqlite_ingestion.py): `ingestion/sqlite_ingestion.py`
- [API Ingestion Runner](https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/scripts/run_api_ingestion.py): `scripts/run_api_ingestion.py`
- [API Landing](https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/ingestion/api_ingestion.py): `ingestion/api_ingestion.py`
- [Raw Data Landing](https://github.com/hsai-rohit/real-time-ecommerce-data-platform/tree/main/data): `data`

### Data Validation

- [CSV Contract Checks](https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/validation/csv_validation.py): `validation/csv_validation.py`
- [Orders & Payments Checks](https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/validation/sqlite_validation.py): `validation/sqlite_validation.py`
- [Events API Checks](https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/validation/api_validation.py): `validation/api_validation.py`

### Cloud Storage

- [ADLS Upload](https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/ingestion/adls_upload.py): `ingestion/adls_upload.py`
- [ADLS Download](https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/ingestion/adls_download.py): `ingestion/adls_download.py`
- [Databricks Volume Upload](https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/ingestion/databricks_volume_upload.py): `ingestion/databricks_volume_upload.py`

### Local Sources

- [Customer & Product CSV](https://github.com/hsai-rohit/real-time-ecommerce-data-platform/tree/main/data/source/csv): `data/source/csv`
- [Orders & Payments DB](https://github.com/hsai-rohit/real-time-ecommerce-data-platform/tree/main/data/source/sqlite): `data/source/sqlite`
- [Mock Events API](https://github.com/hsai-rohit/real-time-ecommerce-data-platform/blob/main/scripts/mock_events_api.py): `scripts/mock_events_api.py`

### Other components

- **Operator**
- **Azure ADLS Gen2**
- **Databricks Volume**

## Connections

- Operator → CSV Ingestion Runner: runs
- Operator → SQLite Ingestion Runner: runs
- Operator → API Ingestion Runner: runs
- CSV Ingestion Runner → CSV Landing: calls (evidence: `scripts/run_csv_ingestion.py`)
- SQLite Ingestion Runner → SQLite Extraction: calls (evidence: `scripts/run_sqlite_ingestion.py`)
- API Ingestion Runner → API Landing: calls (evidence: `scripts/run_api_ingestion.py`)
- CSV Landing → Raw Data Landing: writes (evidence: `ingestion/csv_ingestion.py`)
- SQLite Extraction → Raw Data Landing: writes (evidence: `ingestion/sqlite_ingestion.py`)
- API Landing → Mock Events API: fetches JSON (evidence: `ingestion/api_ingestion.py`)
- API Landing → Raw Data Landing: writes (evidence: `ingestion/api_ingestion.py`)
- CSV Contract Checks → Raw Data Landing: checks CSV
- Orders & Payments Checks → Raw Data Landing: checks CSV
- Events API Checks → Raw Data Landing: checks JSON
- Events API Checks → CSV Contract Checks: uses result type (evidence: `validation/api_validation.py`)
- Orders & Payments Checks → CSV Contract Checks: uses helpers (evidence: `validation/sqlite_validation.py`)
- Operator → ADLS Upload: runs upload
- ADLS Upload → Azure ADLS Gen2: uploads files (evidence: `ingestion/adls_upload.py`)
- Azure ADLS Gen2 → ADLS Download: downloads prefix (evidence: `ingestion/adls_download.py`)
- Operator → ADLS Download: runs download
- Operator → Databricks Volume Upload: runs upload
- Databricks Volume Upload → Databricks Volume: uploads files (evidence: `ingestion/databricks_volume_upload.py`)