# ttt-huong/big-data architecture

> Architecture diagram of the GitHub repository ttt-huong/big-data, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ttt-huong/big-data
- Repository: https://github.com/ttt-huong/big-data

## Overview

This is a local-first batch ETL lakehouse for image metadata. The principal workflow generates or accepts CSV/Parquet batches, extracts them in chunks, enriches and validates records, routes rejected rows to a DLQ, and idempotently merges clean rows into Delta Lake. The orchestrator selects full, incremental, or backfill behavior and maintains watermark/source state; the loader retries transient write failures. A separate analytics entry point queries the lakehouse and DLQ with DuckDB. The README documents these capabilities; the graph follows sampled source where available.

Main entry points: [`main_pipeline.py`](https://github.com/ttt-huong/big-data/blob/main/main_pipeline.py) (Pipeline Orchestrator), [`pipeline/transform.py`](https://github.com/ttt-huong/big-data/blob/main/pipeline/transform.py) (Enrichment and Routing), [`query_analytics.py`](https://github.com/ttt-huong/big-data/blob/main/query_analytics.py) (Analytics Queries), [`generator/data_generator.py`](https://github.com/ttt-huong/big-data/blob/main/generator/data_generator.py) (Metadata Generator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Source and Ingestion"]
  node_orchestrator["Pipeline Orchestrator<br/>[main_pipeline.py]"]
  node_generator["Metadata Generator<br/>[data_generator.py]"]
  node_raw_batches["CSV / Parquet Batches"]
  node_extract["Chunk Extraction<br/>[extract.py]"]
end

subgraph group_quality["Transform and Quality"]
  node_transform["Enrichment and Routing<br/>[transform.py]"]
  node_validation["Quality Validation<br/>[validation.py]"]
end

subgraph group_storage["Lakehouse and State"]
  node_dlq["Error Parquet DLQ"]
  node_loader["Retrying Delta Loader<br/>[load.py]"]
  node_delta[("Clean Delta Lake")]
  node_watermark["Watermark State"]
end

subgraph group_analytics["Analytics"]
  node_analytics["Analytics Queries<br/>[query_analytics.py]"]
end

node_operator(("Operator"))

node_operator -->|"runs"| node_orchestrator
node_orchestrator -->|"generates source"| node_generator
node_generator -->|"writes batches"| node_raw_batches
node_orchestrator -->|"reads chunks"| node_extract
node_extract -->|"reads"| node_raw_batches
node_orchestrator -->|"transforms chunks"| node_transform
node_transform -->|"validates via"| node_validation
node_transform -->|"writes rejected rows"| node_dlq
node_orchestrator -->|"loads clean rows"| node_loader
node_loader -->|"merges by image_id"| node_delta
node_orchestrator -->|"reads and updates"| node_watermark
node_analytics -->|"queries"| node_delta
node_analytics -->|"counts errors"| node_dlq

click node_orchestrator "https://github.com/ttt-huong/big-data/blob/main/main_pipeline.py"
click node_generator "https://github.com/ttt-huong/big-data/blob/main/generator/data_generator.py"
click node_raw_batches "https://github.com/ttt-huong/big-data/tree/main/data"
click node_extract "https://github.com/ttt-huong/big-data/blob/main/pipeline/extract.py"
click node_transform "https://github.com/ttt-huong/big-data/blob/main/pipeline/transform.py"
click node_validation "https://github.com/ttt-huong/big-data/blob/main/pipeline/validation.py"
click node_dlq "https://github.com/ttt-huong/big-data/tree/main/data"
click node_loader "https://github.com/ttt-huong/big-data/blob/main/pipeline/load.py"
click node_delta "https://github.com/ttt-huong/big-data/tree/main/data"
click node_watermark "https://github.com/ttt-huong/big-data/tree/main/data"
click node_analytics "https://github.com/ttt-huong/big-data/blob/main/query_analytics.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_orchestrator,node_generator,node_raw_batches,node_extract toneBlue
class node_transform,node_validation toneAmber
class node_dlq,node_loader,node_delta,node_watermark toneMint
class node_analytics toneRose
class node_operator toneIndigo
```

## Components

### Source and Ingestion

- [Pipeline Orchestrator](https://github.com/ttt-huong/big-data/blob/main/main_pipeline.py): `main_pipeline.py`
- [Metadata Generator](https://github.com/ttt-huong/big-data/blob/main/generator/data_generator.py): `generator/data_generator.py`
- [CSV / Parquet Batches](https://github.com/ttt-huong/big-data/tree/main/data): `data`
- [Chunk Extraction](https://github.com/ttt-huong/big-data/blob/main/pipeline/extract.py): `pipeline/extract.py`

### Transform and Quality

- [Enrichment and Routing](https://github.com/ttt-huong/big-data/blob/main/pipeline/transform.py): `pipeline/transform.py`
- [Quality Validation](https://github.com/ttt-huong/big-data/blob/main/pipeline/validation.py): `pipeline/validation.py`

### Lakehouse and State

- [Error Parquet DLQ](https://github.com/ttt-huong/big-data/tree/main/data): `data`
- [Retrying Delta Loader](https://github.com/ttt-huong/big-data/blob/main/pipeline/load.py): `pipeline/load.py`
- [Clean Delta Lake](https://github.com/ttt-huong/big-data/tree/main/data): `data`
- [Watermark State](https://github.com/ttt-huong/big-data/tree/main/data): `data`

### Analytics

- [Analytics Queries](https://github.com/ttt-huong/big-data/blob/main/query_analytics.py): `query_analytics.py`

### Other components

- **Operator**

## Connections

- Operator → Pipeline Orchestrator: runs
- Pipeline Orchestrator → Metadata Generator: generates source
- Metadata Generator → CSV / Parquet Batches: writes batches
- Pipeline Orchestrator → Chunk Extraction: reads chunks
- Chunk Extraction → CSV / Parquet Batches: reads
- Pipeline Orchestrator → Enrichment and Routing: transforms chunks
- Enrichment and Routing → Quality Validation: validates via
- Enrichment and Routing → Error Parquet DLQ: writes rejected rows
- Pipeline Orchestrator → Retrying Delta Loader: loads clean rows
- Retrying Delta Loader → Clean Delta Lake: merges by image_id
- Pipeline Orchestrator → Watermark State: reads and updates
- Analytics Queries → Clean Delta Lake: queries
- Analytics Queries → Error Parquet DLQ: counts errors