# lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering architecture

> Architecture diagram of the GitHub repository lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering
- Repository: https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering

## Overview

This repository is a data-ingestion engineering lab, with a PostgreSQL/pgvector starter pipeline and a benchmark workflow—not a chatbot. The principal workflow loads and normalizes PDF pages, chunks them under three strategies, embeds and indexes them, then retrieves evidence for benchmark questions and records evaluation traces and summaries. Manual audits add chunk-coherence, metadata, and grounding review. The graph covers all seven core starter modules. The run.py excerpt is partial, so links for unsampled benchmark inputs and some orchestration details are omitted.

Main entry points: [`starter/src/run.py`](https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/starter/src/run.py) (Pipeline runner), [`starter/src/store.py`](https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/starter/src/store.py) (Vector store), [`starter/src/ingestion.py`](https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/starter/src/ingestion.py) (PDF ingestion), [`starter/src/models.py`](https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/starter/src/models.py) (Page and chunk models)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Ingestion and chunking"]
  node_ingestion["PDF ingestion<br/>[ingestion.py]"]
  node_models["Page and chunk models<br/>[models.py]"]
  node_chunking["Chunk strategies<br/>[chunking.py]"]
end

subgraph group_index["Embedding and storage"]
  node_embedder["Embedding adapter<br/>[embeddings.py]"]
  node_store["Vector store<br/>[store.py]"]
end

subgraph group_evaluation["Retrieval and evaluation"]
  node_runner["Pipeline runner<br/>[run.py]"]
  node_evaluation["Retrieval evaluation<br/>[evaluation.py]"]
end

subgraph group_outputs["Audit and results"]
  node_audit["Manual audit sampling<br/>[run.py]"]
  node_results["Pipeline results"]
end

node_student(("Lab participant"))
node_corpus["Source corpus<br/>[manifest.json]"]
node_questions["Benchmark questions<br/>[questions.json]"]
node_pgvector[("PostgreSQL pgvector")]
node_providers{{"Embedding APIs"}}

node_student -->|"runs pipeline"| node_runner
node_corpus -->|"documents workflow"| node_ingestion
node_runner -->|"loads pages"| node_ingestion
node_ingestion -->|"creates pages"| node_models
node_runner -->|"builds strategies"| node_chunking
node_chunking -->|"creates chunks"| node_models
node_runner -->|"embeds chunks"| node_embedder
node_embedder -->|"requests vectors"| node_providers
node_runner -->|"upserts and searches"| node_store
node_store -->|"reads chunks"| node_models
node_store -->|"persists and searches"| node_pgvector
node_runner -->|"evaluates retrieved hits"| node_evaluation
node_questions -.->|"benchmark input"| node_runner
node_runner -->|"builds audit samples"| node_audit
node_runner -->|"writes traces and summaries"| node_results
node_student -.->|"reviews samples"| node_audit

click node_corpus "https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/data/manifest.json"
click node_questions "https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/benchmark/questions.json"
click node_runner "https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/starter/src/run.py"
click node_ingestion "https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/starter/src/ingestion.py"
click node_models "https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/starter/src/models.py"
click node_chunking "https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/starter/src/chunking.py"
click node_embedder "https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/starter/src/embeddings.py"
click node_store "https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/starter/src/store.py"
click node_evaluation "https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/starter/src/evaluation.py"
click node_audit "https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/starter/src/run.py"
click node_results "https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/tree/main/results"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingestion,node_models,node_chunking toneBlue
class node_embedder,node_store,node_pgvector toneAmber
class node_runner,node_evaluation,node_providers toneMint
class node_audit,node_results toneRose
class node_student toneIndigo
class node_corpus,node_questions toneTeal
```

## Components

### Ingestion and chunking

- [PDF ingestion](https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/starter/src/ingestion.py): `starter/src/ingestion.py`
- [Page and chunk models](https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/starter/src/models.py): `starter/src/models.py`
- [Chunk strategies](https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/starter/src/chunking.py): `starter/src/chunking.py`

### Embedding and storage

- [Embedding adapter](https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/starter/src/embeddings.py): `starter/src/embeddings.py`
- [Vector store](https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/starter/src/store.py): `starter/src/store.py`

### Retrieval and evaluation

- [Pipeline runner](https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/starter/src/run.py): `starter/src/run.py`
- [Retrieval evaluation](https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/starter/src/evaluation.py): `starter/src/evaluation.py`

### Audit and results

- [Manual audit sampling](https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/starter/src/run.py): `starter/src/run.py`
- [Pipeline results](https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/tree/main/results): `results`

### Other components

- **Lab participant**
- [Source corpus](https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/data/manifest.json): `data/manifest.json`
- [Benchmark questions](https://github.com/lyslama/k5-l3a-day06-2a202603475-lythanhlam-dataingestionengineering/blob/main/benchmark/questions.json): `benchmark/questions.json`
- **PostgreSQL pgvector**
- **Embedding APIs**

## Connections

- Lab participant → Pipeline runner: runs pipeline (evidence: `README.md`)
- Source corpus → PDF ingestion: documents workflow (evidence: `README.md`)
- Pipeline runner → PDF ingestion: loads pages (evidence: `starter/src/run.py`)
- PDF ingestion → Page and chunk models: creates pages (evidence: `starter/src/ingestion.py`)
- Pipeline runner → Chunk strategies: builds strategies (evidence: `starter/src/run.py`)
- Chunk strategies → Page and chunk models: creates chunks (evidence: `starter/src/chunking.py`)
- Pipeline runner → Embedding adapter: embeds chunks (evidence: `starter/src/run.py`)
- Embedding adapter → Embedding APIs: requests vectors (evidence: `starter/src/embeddings.py`)
- Pipeline runner → Vector store: upserts and searches (evidence: `starter/src/run.py`)
- Vector store → Page and chunk models: reads chunks (evidence: `starter/src/store.py`)
- Vector store → PostgreSQL pgvector: persists and searches (evidence: `starter/src/store.py`)
- Pipeline runner → Retrieval evaluation: evaluates retrieved hits (evidence: `starter/src/run.py`)
- Benchmark questions → Pipeline runner: benchmark input
- Pipeline runner → Manual audit sampling: builds audit samples (evidence: `starter/src/run.py`)
- Pipeline runner → Pipeline results: writes traces and summaries (evidence: `starter/src/run.py`)
- Lab participant → Manual audit sampling: reviews samples (evidence: `README.md`)