How vinuni-ai20k/k5-l3-day06-dataingestionengineering works
This repository is a data-ingestion engineering lab, not a completed application: its starter modules are largely TODOs. The intended workflow turns source documents into traceable chunks and vectors, searches them against benchmark questions, and evaluates results across configurations. The graph combines that documented workflow with the starter’s module contracts; links between stages are supported by README guidance or the suggested run sequence, not by completed runtime calls.
Main entry points: run.py Pipeline runner · ingestion.py Page loading · chunking.py Chunk strategies · embeddings.py Embedding provider
Components 12
Ingestion
- Source corpus
- Page loading
starter/src/ingestion.py - Page and chunk models
starter/src/models.py - Chunk strategies
starter/src/chunking.py - Embedding provider
starter/src/embeddings.py
Retrieval
- Vector store
starter/src/store.py
Evaluation
- Benchmark questions
- Pipeline runner
starter/src/run.py - Retrieval evaluation
starter/src/evaluation.py - Traces and results
starter/results - Manual assessment
Other components
- Student
Connections 16
- Student → Source corpus: selects
README.md - Source corpus → Page loading: provides documents
README.md - Page loading → Page and chunk models: creates pages
starter/src/ingestion.py - Page and chunk models → Chunk strategies: provides pages
starter/src/chunking.py - Chunk strategies → Embedding provider: produces chunks
README.md - Embedding provider → Vector store: supplies vectors
README.md - Student → Benchmark questions: prepares questions
README.md - Pipeline runner → Page loading: loads corpus
starter/src/run.py - Pipeline runner → Chunk strategies: builds strategies
starter/src/run.py - Pipeline runner → Embedding provider: embeds chunks
starter/src/run.py - Pipeline runner → Vector store: stores and searches
starter/src/run.py - Pipeline runner → Benchmark questions: runs questions
starter/src/run.py - Pipeline runner → Retrieval evaluation: evaluates hits
starter/src/run.py - Retrieval evaluation → Traces and results: records traces
README.md - Pipeline runner → Traces and results: saves top-three traces
starter/src/run.py - Traces and results → Manual assessment: provides samples
README.md