# minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion architecture

> Architecture diagram of the GitHub repository minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion
- Repository: https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion

## Overview

This repository is a data-ingestion engineering lab with a working document-to-vector retrieval benchmark. Its main workflow verifies corpus pages and benchmark questions, creates alternative chunks, embeds and indexes them, retrieves top results, then evaluates and records traces and metrics. The README also documents manual audits and comparison across three strategies. The graph reflects the sampled runtime and documented capabilities; some internal details of the run orchestration and evaluation are unsampled.

Main entry points: [`starter/src/run.py`](https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/starter/src/run.py) (Pipeline runner), [`starter/src/ingestion.py`](https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/starter/src/ingestion.py) (Corpus verification), [`starter/scripts/prepare_corpus.py`](https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/starter/scripts/prepare_corpus.py) (Corpus extraction), [`starter/src/models.py`](https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/starter/src/models.py) (Page and chunk models)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_corpus["Corpus Preparation"]
  node_prep["Corpus extraction<br/>[prepare_corpus.py]"]
  node_corpusdata["Corpus manifest<br/>[manifest.json]"]
end

subgraph group_pipeline["Retrieval Pipeline"]
  node_loader["Corpus verification<br/>[ingestion.py]"]
  node_contracts["Page and chunk models<br/>[models.py]"]
  node_chunking["Chunk strategies<br/>[chunking.py]"]
  node_embedder["Local embeddings<br/>[embeddings.py]"]
  node_runner["Pipeline runner<br/>[run.py]"]
end

subgraph group_benchmark["Benchmarking"]
  node_benchmarkdata["Benchmark questions<br/>[benchmark.json]"]
  node_evaluation["Retrieval evaluation<br/>[evaluation.py]"]
  node_audit["Manual audit candidates"]
  node_results["Run results"]
end

subgraph group_infra["Vector Infrastructure"]
  node_store["Qdrant store adapter<br/>[store.py]"]
end

node_operator(("Lab operator"))
node_sourcepdf["Source PDF"]
node_qdrant[("Qdrant server")]

node_operator -->|"prepares corpus"| node_prep
node_sourcepdf -->|"extracts pages"| node_prep
node_prep -->|"writes snapshot"| node_corpusdata
node_operator -->|"runs benchmark"| node_runner
node_runner -->|"loads corpus"| node_loader
node_loader -->|"verifies and reads"| node_corpusdata
node_runner -->|"reads questions"| node_benchmarkdata
node_loader -->|"creates pages"| node_contracts
node_runner -->|"creates chunks"| node_chunking
node_chunking -->|"uses page and chunk types"| node_contracts
node_runner -->|"embeds chunks and queries"| node_embedder
node_runner -->|"indexes and searches"| node_store
node_store -->|"writes and queries"| node_qdrant
node_runner -->|"validates and scores"| node_evaluation
node_evaluation -->|"checks source pages"| node_contracts
node_runner -->|"generates candidates"| node_audit
node_runner -->|"records traces and metrics"| node_results

click node_prep "https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/starter/scripts/prepare_corpus.py"
click node_corpusdata "https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/data/manifest.json"
click node_loader "https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/starter/src/ingestion.py"
click node_contracts "https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/starter/src/models.py"
click node_chunking "https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/starter/src/chunking.py"
click node_embedder "https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/starter/src/embeddings.py"
click node_runner "https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/starter/src/run.py"
click node_benchmarkdata "https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/data/benchmark.json"
click node_evaluation "https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/starter/src/evaluation.py"
click node_audit "https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/starter/results/final_v3/audit_candidates.json"
click node_results "https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/tree/main/starter/results/final_v3"
click node_store "https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/starter/src/store.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_prep,node_corpusdata toneBlue
class node_loader,node_contracts,node_chunking,node_embedder,node_runner,node_qdrant toneAmber
class node_benchmarkdata,node_evaluation,node_audit,node_results toneMint
class node_store toneRose
class node_operator,node_sourcepdf toneIndigo
```

## Components

### Corpus Preparation

- [Corpus extraction](https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/starter/scripts/prepare_corpus.py): `starter/scripts/prepare_corpus.py`
- [Corpus manifest](https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/data/manifest.json): `data/manifest.json`

### Retrieval Pipeline

- [Corpus verification](https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/starter/src/ingestion.py): `starter/src/ingestion.py`
- [Page and chunk models](https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/starter/src/models.py): `starter/src/models.py`
- [Chunk strategies](https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/starter/src/chunking.py): `starter/src/chunking.py`
- [Local embeddings](https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/starter/src/embeddings.py): `starter/src/embeddings.py`
- [Pipeline runner](https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/starter/src/run.py): `starter/src/run.py`

### Benchmarking

- [Benchmark questions](https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/data/benchmark.json): `data/benchmark.json`
- [Retrieval evaluation](https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/starter/src/evaluation.py): `starter/src/evaluation.py`
- [Manual audit candidates](https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/starter/results/final_v3/audit_candidates.json): `starter/results/final_v3/audit_candidates.json`
- [Run results](https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/tree/main/starter/results/final_v3): `starter/results/final_v3`

### Vector Infrastructure

- [Qdrant store adapter](https://github.com/minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion/blob/main/starter/src/store.py): `starter/src/store.py`

### Other components

- **Lab operator**
- **Source PDF**
- **Qdrant server**

## Connections

- Lab operator → Corpus extraction: prepares corpus (evidence: `README.md`)
- Source PDF → Corpus extraction: extracts pages (evidence: `starter/scripts/prepare_corpus.py`)
- Corpus extraction → Corpus manifest: writes snapshot (evidence: `starter/scripts/prepare_corpus.py`)
- Lab operator → Pipeline runner: runs benchmark (evidence: `README.md`)
- Pipeline runner → Corpus verification: loads corpus (evidence: `starter/src/run.py`)
- Corpus verification → Corpus manifest: verifies and reads (evidence: `starter/src/ingestion.py`)
- Pipeline runner → Benchmark questions: reads questions (evidence: `starter/src/run.py`)
- Corpus verification → Page and chunk models: creates pages (evidence: `starter/src/ingestion.py`)
- Pipeline runner → Chunk strategies: creates chunks (evidence: `starter/src/run.py`)
- Chunk strategies → Page and chunk models: uses page and chunk types (evidence: `starter/src/chunking.py`)
- Pipeline runner → Local embeddings: embeds chunks and queries (evidence: `starter/src/run.py`)
- Pipeline runner → Qdrant store adapter: indexes and searches (evidence: `starter/src/run.py`)
- Qdrant store adapter → Qdrant server: writes and queries (evidence: `starter/src/store.py`)
- Pipeline runner → Retrieval evaluation: validates and scores (evidence: `starter/src/run.py`)
- Retrieval evaluation → Page and chunk models: checks source pages (evidence: `starter/src/evaluation.py`)
- Pipeline runner → Manual audit candidates: generates candidates (evidence: `README.md`)
- Pipeline runner → Run results: records traces and metrics (evidence: `README.md`)