How autonoma-tools/how-to-test-rag-retrieval works
This repository is an offline harness for evaluating RAG retrieval without generation, network access, or a vector database. The main workflow embeds a query, returns ranked chunk IDs, and scores those results against labeled query-to-chunk pairs using hit rate, MRR, context precision, and context recall. The default corpus and deterministic embedder are lightweight internal layers supporting that workflow. The README documents the runnable report and regression gate; tests and CI machinery are omitted from this runtime map.
Main entry points: run_retrieval.py Retrieval report demo · mrr.py Mean reciprocal rank · hit_rate.py Hit rate · retriever.py Retriever
Components 12
Retrieval
- Retriever
src/retriever.py - Help corpus
src/corpus.py - Hashing embedder
src/embedding.py - Ranked chunk IDs
Evaluation
- Hit rate
src/metrics/hit_rate.py - Mean reciprocal rank
src/metrics/mrr.py - Context precision and recall
src/metrics/context_precision_recall.py - Metric input guards
src/metrics/common.py
Evaluation data
- Labeled query pairs
tests/fixtures/labeled_queries.json
Reporting
- Retrieval report demo
examples/run_retrieval.py - Aggregate metric results
Other components
- Engineer
Connections 20
- Engineer → Retrieval report demo: runs
README.md - Retrieval report demo → Retriever: queries
README.md - Retriever → Help corpus: loads chunks
src/retriever.py - Retriever → Hashing embedder: embeds text
src/retriever.py - Retriever → Ranked chunk IDs: returns
src/retriever.py - Retrieval report demo → Labeled query pairs: scores against
README.md - Retrieval report demo → Hit rate: computes
README.md - Retrieval report demo → Mean reciprocal rank: computes
README.md - Retrieval report demo → Context precision and recall: computes
README.md - Ranked chunk IDs → Hit rate: scores
README.md - Ranked chunk IDs → Mean reciprocal rank: scores
README.md - Ranked chunk IDs → Context precision and recall: scores
README.md - Hit rate → Metric input guards: uses
src/metrics/hit_rate.py - Mean reciprocal rank → Metric input guards: uses
src/metrics/mrr.py - Context precision and recall → Metric input guards: uses
src/metrics/context_precision_recall.py - Mean reciprocal rank → Hit rate: delegates mean
src/metrics/mrr.py - Hit rate → Aggregate metric results: produces scores
README.md - Mean reciprocal rank → Aggregate metric results: produces scores
README.md - Context precision and recall → Aggregate metric results: produces scores
README.md - Retrieval report demo → Aggregate metric results: prints
README.md