# vinuni-ai20k/k5-l3-day06-dataingestionengineering architecture

> Architecture diagram of the GitHub repository vinuni-ai20k/k5-l3-day06-dataingestionengineering, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vinuni-ai20k/k5-l3-day06-dataingestionengineering
- Repository: https://github.com/vinuni-ai20k/k5-l3-day06-dataingestionengineering

## Overview

This repository is a data-ingestion engineering lab, not a completed application: its starter modules are largely TODOs. The intended workflow turns source documents into traceable chunks and vectors, searches them against benchmark questions, and evaluates results across configurations. The graph combines that documented workflow with the starter’s module contracts; links between stages are supported by README guidance or the suggested run sequence, not by completed runtime calls.

Main entry points: [`starter/src/run.py`](https://github.com/vinuni-ai20k/k5-l3-day06-dataingestionengineering/blob/main/starter/src/run.py) (Pipeline runner), [`starter/src/ingestion.py`](https://github.com/vinuni-ai20k/k5-l3-day06-dataingestionengineering/blob/main/starter/src/ingestion.py) (Page loading), [`starter/src/chunking.py`](https://github.com/vinuni-ai20k/k5-l3-day06-dataingestionengineering/blob/main/starter/src/chunking.py) (Chunk strategies), [`starter/src/embeddings.py`](https://github.com/vinuni-ai20k/k5-l3-day06-dataingestionengineering/blob/main/starter/src/embeddings.py) (Embedding provider)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Ingestion"]
  node_corpus["Source corpus"]
  node_page_loader["Page loading<br/>[ingestion.py]"]
  node_page_chunk["Page and chunk models<br/>[models.py]"]
  node_chunking["Chunk strategies<br/>[chunking.py]"]
  node_embedder["Embedding provider<br/>[embeddings.py]"]
end

subgraph group_retrieval["Retrieval"]
  node_vector_store[("Vector store<br/>[store.py]")]
end

subgraph group_evaluation["Evaluation"]
  node_benchmark_questions["Benchmark questions"]
  node_runner["Pipeline runner<br/>[run.py]"]
  node_retrieval_eval["Retrieval evaluation<br/>[evaluation.py]"]
  node_results["Traces and results"]
  node_manual_review["Manual assessment"]
end

node_student(("Student"))

node_student -->|"selects"| node_corpus
node_corpus -->|"provides documents"| node_page_loader
node_page_loader -->|"creates pages"| node_page_chunk
node_page_chunk -->|"provides pages"| node_chunking
node_chunking -->|"produces chunks"| node_embedder
node_embedder -->|"supplies vectors"| node_vector_store
node_student -->|"prepares questions"| node_benchmark_questions
node_runner -->|"loads corpus"| node_page_loader
node_runner -->|"builds strategies"| node_chunking
node_runner -->|"embeds chunks"| node_embedder
node_runner -->|"stores and searches"| node_vector_store
node_runner -->|"runs questions"| node_benchmark_questions
node_runner -->|"evaluates hits"| node_retrieval_eval
node_retrieval_eval -->|"records traces"| node_results
node_runner -->|"saves top-three traces"| node_results
node_results -->|"provides samples"| node_manual_review

click node_page_loader "https://github.com/vinuni-ai20k/k5-l3-day06-dataingestionengineering/blob/main/starter/src/ingestion.py"
click node_page_chunk "https://github.com/vinuni-ai20k/k5-l3-day06-dataingestionengineering/blob/main/starter/src/models.py"
click node_chunking "https://github.com/vinuni-ai20k/k5-l3-day06-dataingestionengineering/blob/main/starter/src/chunking.py"
click node_embedder "https://github.com/vinuni-ai20k/k5-l3-day06-dataingestionengineering/blob/main/starter/src/embeddings.py"
click node_vector_store "https://github.com/vinuni-ai20k/k5-l3-day06-dataingestionengineering/blob/main/starter/src/store.py"
click node_runner "https://github.com/vinuni-ai20k/k5-l3-day06-dataingestionengineering/blob/main/starter/src/run.py"
click node_retrieval_eval "https://github.com/vinuni-ai20k/k5-l3-day06-dataingestionengineering/blob/main/starter/src/evaluation.py"
click node_results "https://github.com/vinuni-ai20k/k5-l3-day06-dataingestionengineering/tree/main/starter/results"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_corpus,node_page_loader,node_page_chunk,node_chunking,node_embedder toneBlue
class node_vector_store toneAmber
class node_benchmark_questions,node_runner,node_retrieval_eval,node_results,node_manual_review toneMint
class node_student toneIndigo
```

## Components

### Ingestion

- **Source corpus**
- [Page loading](https://github.com/vinuni-ai20k/k5-l3-day06-dataingestionengineering/blob/main/starter/src/ingestion.py): `starter/src/ingestion.py`
- [Page and chunk models](https://github.com/vinuni-ai20k/k5-l3-day06-dataingestionengineering/blob/main/starter/src/models.py): `starter/src/models.py`
- [Chunk strategies](https://github.com/vinuni-ai20k/k5-l3-day06-dataingestionengineering/blob/main/starter/src/chunking.py): `starter/src/chunking.py`
- [Embedding provider](https://github.com/vinuni-ai20k/k5-l3-day06-dataingestionengineering/blob/main/starter/src/embeddings.py): `starter/src/embeddings.py`

### Retrieval

- [Vector store](https://github.com/vinuni-ai20k/k5-l3-day06-dataingestionengineering/blob/main/starter/src/store.py): `starter/src/store.py`

### Evaluation

- **Benchmark questions**
- [Pipeline runner](https://github.com/vinuni-ai20k/k5-l3-day06-dataingestionengineering/blob/main/starter/src/run.py): `starter/src/run.py`
- [Retrieval evaluation](https://github.com/vinuni-ai20k/k5-l3-day06-dataingestionengineering/blob/main/starter/src/evaluation.py): `starter/src/evaluation.py`
- [Traces and results](https://github.com/vinuni-ai20k/k5-l3-day06-dataingestionengineering/tree/main/starter/results): `starter/results`
- **Manual assessment**

### Other components

- **Student**

## Connections

- Student → Source corpus: selects (evidence: `README.md`)
- Source corpus → Page loading: provides documents (evidence: `README.md`)
- Page loading → Page and chunk models: creates pages (evidence: `starter/src/ingestion.py`)
- Page and chunk models → Chunk strategies: provides pages (evidence: `starter/src/chunking.py`)
- Chunk strategies → Embedding provider: produces chunks (evidence: `README.md`)
- Embedding provider → Vector store: supplies vectors (evidence: `README.md`)
- Student → Benchmark questions: prepares questions (evidence: `README.md`)
- Pipeline runner → Page loading: loads corpus (evidence: `starter/src/run.py`)
- Pipeline runner → Chunk strategies: builds strategies (evidence: `starter/src/run.py`)
- Pipeline runner → Embedding provider: embeds chunks (evidence: `starter/src/run.py`)
- Pipeline runner → Vector store: stores and searches (evidence: `starter/src/run.py`)
- Pipeline runner → Benchmark questions: runs questions (evidence: `starter/src/run.py`)
- Pipeline runner → Retrieval evaluation: evaluates hits (evidence: `starter/src/run.py`)
- Retrieval evaluation → Traces and results: records traces (evidence: `README.md`)
- Pipeline runner → Traces and results: saves top-three traces (evidence: `starter/src/run.py`)
- Traces and results → Manual assessment: provides samples (evidence: `README.md`)