# uzankhan/retail-order-etl architecture

> Architecture diagram of the GitHub repository uzankhan/retail-order-etl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/uzankhan/retail-order-etl
- Repository: https://github.com/uzankhan/retail-order-etl

## Overview

This is a batch retail-order ETL pipeline. An operator runs it for a bucket, region, and batch date; it reads raw orders from S3, deduplicates and validates rows, calculates revenue, loads accepted orders into PostgreSQL, verifies database totals, and publishes sales, rejection, and metrics reports to S3. The transformation and reporting stages are documented and visible in pipeline.py. Cloud networking and IAM details are described in the README, but their infrastructure implementation is not present in the sampled source.

Main entry points: [`pipeline.py`](https://github.com/uzankhan/retail-order-etl/blob/main/pipeline.py) (Pipeline entry point)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Batch ingestion"]
  node_runner["Pipeline entry point<br/>[pipeline.py]"]
  node_s3[("Amazon S3")]
  node_extract["CSV extraction<br/>[pipeline.py]"]
end

subgraph group_transform["Order transformation"]
  node_dedup["Order deduplication<br/>[pipeline.py]"]
  node_validate["Business validation<br/>[pipeline.py]"]
  node_revenue["Revenue calculation<br/>[pipeline.py]"]
end

subgraph group_warehouse["Warehouse loading"]
  node_connect["TLS database connection<br/>[pipeline.py]"]
  node_orders[("PostgreSQL orders")]
  node_load["Idempotent order load<br/>[pipeline.py]"]
  node_verify["Database totals<br/>[pipeline.py]"]
end

subgraph group_reporting["Reporting"]
  node_aggregate["Daily sales aggregation<br/>[pipeline.py]"]
  node_reports["Report publishing<br/>[pipeline.py]"]
end

node_operator(("Operator"))

node_operator -->|"runs batch"| node_runner
node_runner -->|"calls"| node_extract
node_extract -->|"reads raw CSV"| node_s3
node_runner -->|"calls transform"| node_dedup
node_dedup -->|"removes duplicates before"| node_validate
node_validate -->|"filters before"| node_revenue
node_runner -->|"opens connection"| node_connect
node_runner -->|"loads accepted rows"| node_load
node_load -->|"inserts rows"| node_orders
node_runner -->|"queries totals"| node_verify
node_verify -->|"reads totals"| node_orders
node_runner -->|"aggregates accepted rows"| node_aggregate
node_runner -->|"publishes outputs"| node_reports
node_reports -->|"writes reports"| node_s3
node_aggregate -->|"provides sales summary"| node_reports

click node_runner "https://github.com/uzankhan/retail-order-etl/blob/main/pipeline.py"
click node_extract "https://github.com/uzankhan/retail-order-etl/blob/main/pipeline.py"
click node_dedup "https://github.com/uzankhan/retail-order-etl/blob/main/pipeline.py"
click node_validate "https://github.com/uzankhan/retail-order-etl/blob/main/pipeline.py"
click node_revenue "https://github.com/uzankhan/retail-order-etl/blob/main/pipeline.py"
click node_connect "https://github.com/uzankhan/retail-order-etl/blob/main/pipeline.py"
click node_load "https://github.com/uzankhan/retail-order-etl/blob/main/pipeline.py"
click node_verify "https://github.com/uzankhan/retail-order-etl/blob/main/pipeline.py"
click node_aggregate "https://github.com/uzankhan/retail-order-etl/blob/main/pipeline.py"
click node_reports "https://github.com/uzankhan/retail-order-etl/blob/main/pipeline.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner,node_s3,node_extract toneBlue
class node_dedup,node_validate,node_revenue toneAmber
class node_connect,node_orders,node_load,node_verify toneMint
class node_aggregate,node_reports toneRose
class node_operator toneIndigo
```

## Components

### Batch ingestion

- [Pipeline entry point](https://github.com/uzankhan/retail-order-etl/blob/main/pipeline.py): `pipeline.py`
- **Amazon S3**
- [CSV extraction](https://github.com/uzankhan/retail-order-etl/blob/main/pipeline.py): `pipeline.py`

### Order transformation

- [Order deduplication](https://github.com/uzankhan/retail-order-etl/blob/main/pipeline.py): `pipeline.py`
- [Business validation](https://github.com/uzankhan/retail-order-etl/blob/main/pipeline.py): `pipeline.py`
- [Revenue calculation](https://github.com/uzankhan/retail-order-etl/blob/main/pipeline.py): `pipeline.py`

### Warehouse loading

- [TLS database connection](https://github.com/uzankhan/retail-order-etl/blob/main/pipeline.py): `pipeline.py`
- **PostgreSQL orders**
- [Idempotent order load](https://github.com/uzankhan/retail-order-etl/blob/main/pipeline.py): `pipeline.py`
- [Database totals](https://github.com/uzankhan/retail-order-etl/blob/main/pipeline.py): `pipeline.py`

### Reporting

- [Daily sales aggregation](https://github.com/uzankhan/retail-order-etl/blob/main/pipeline.py): `pipeline.py`
- [Report publishing](https://github.com/uzankhan/retail-order-etl/blob/main/pipeline.py): `pipeline.py`

### Other components

- **Operator**

## Connections

- Operator → Pipeline entry point: runs batch (evidence: `README.md`)
- Pipeline entry point → CSV extraction: calls (evidence: `pipeline.py`)
- CSV extraction → Amazon S3: reads raw CSV (evidence: `pipeline.py`)
- Pipeline entry point → Order deduplication: calls transform (evidence: `pipeline.py`)
- Order deduplication → Business validation: removes duplicates before (evidence: `pipeline.py`)
- Business validation → Revenue calculation: filters before (evidence: `pipeline.py`)
- Pipeline entry point → TLS database connection: opens connection (evidence: `pipeline.py`)
- Pipeline entry point → Idempotent order load: loads accepted rows (evidence: `pipeline.py`)
- Idempotent order load → PostgreSQL orders: inserts rows (evidence: `pipeline.py`)
- Pipeline entry point → Database totals: queries totals (evidence: `pipeline.py`)
- Database totals → PostgreSQL orders: reads totals (evidence: `pipeline.py`)
- Pipeline entry point → Daily sales aggregation: aggregates accepted rows (evidence: `pipeline.py`)
- Pipeline entry point → Report publishing: publishes outputs (evidence: `pipeline.py`)
- Report publishing → Amazon S3: writes reports (evidence: `pipeline.py`)
- Daily sales aggregation → Report publishing: provides sales summary (evidence: `pipeline.py`)
