How minhktm02/k5-l3a-khuattienminh-day06-2a202603616-dataingestion works
This repository is a data-ingestion engineering lab with a working document-to-vector retrieval benchmark. Its main workflow verifies corpus pages and benchmark questions, creates alternative chunks, embeds and indexes them, retrieves top results, then evaluates and records traces and metrics. The README also documents manual audits and comparison across three strategies. The graph reflects the sampled runtime and documented capabilities; some internal details of the run orchestration and evaluation are unsampled.
Main entry points: run.py Pipeline runner · ingestion.py Corpus verification · prepare_corpus.py Corpus extraction · models.py Page and chunk models
Components 15
Corpus Preparation
- Corpus extraction
starter/scripts/prepare_corpus.py - Corpus manifest
data/manifest.json
Retrieval Pipeline
- Corpus verification
starter/src/ingestion.py - Page and chunk models
starter/src/models.py - Chunk strategies
starter/src/chunking.py - Local embeddings
starter/src/embeddings.py - Pipeline runner
starter/src/run.py
Benchmarking
- Benchmark questions
data/benchmark.json - Retrieval evaluation
starter/src/evaluation.py - Manual audit candidates
starter/results/final_v3/audit_candidates.json - Run results
starter/results/final_v3
Vector Infrastructure
- Qdrant store adapter
starter/src/store.py
Other components
- Lab operator
- Source PDF
- Qdrant server
Connections 17
- Lab operator → Corpus extraction: prepares corpus
README.md - Source PDF → Corpus extraction: extracts pages
starter/scripts/prepare_corpus.py - Corpus extraction → Corpus manifest: writes snapshot
starter/scripts/prepare_corpus.py - Lab operator → Pipeline runner: runs benchmark
README.md - Pipeline runner → Corpus verification: loads corpus
starter/src/run.py - Corpus verification → Corpus manifest: verifies and reads
starter/src/ingestion.py - Pipeline runner → Benchmark questions: reads questions
starter/src/run.py - Corpus verification → Page and chunk models: creates pages
starter/src/ingestion.py - Pipeline runner → Chunk strategies: creates chunks
starter/src/run.py - Chunk strategies → Page and chunk models: uses page and chunk types
starter/src/chunking.py - Pipeline runner → Local embeddings: embeds chunks and queries
starter/src/run.py - Pipeline runner → Qdrant store adapter: indexes and searches
starter/src/run.py - Qdrant store adapter → Qdrant server: writes and queries
starter/src/store.py - Pipeline runner → Retrieval evaluation: validates and scores
starter/src/run.py - Retrieval evaluation → Page and chunk models: checks source pages
starter/src/evaluation.py - Pipeline runner → Manual audit candidates: generates candidates
README.md - Pipeline runner → Run results: records traces and metrics
README.md