How mghayaz/book_rag works
This repository implements book-question retrieval: PDFs are extracted and chunked, chunks are embedded into a persistent Chroma store, and queries can retrieve candidates and rerank them. A separate evaluation script runs retrieval against a golden dataset using contextual recall and precision metrics. There is no README or application/API layer shown, so the user-facing entry point and any serving workflow are uncertain; the graph focuses on the evidenced retrieval and evaluation paths.
Main entry points: retriever.py Vector retrieval · reranker.py Cross-encoder reranker · eval_retriever.py Evaluation runner · chorma_book_store Persistent vector store
Components 13
Book ingestion
- Book PDFs
Books - PDF extraction
src/retriever.py - Chunk and embed
src/retriever.py - Persistent vector store
chorma_book_store
Search and ranking
- Vector retrieval
src/retriever.py - Cross-encoder reranker
src/reranker.py
Configuration
- Runtime settings
core/config.py
Retrieval evaluation
- Evaluation runner
evals/eval_retriever.py - Golden queries
goldens/dataset.json - Contextual metrics
evals/eval_retriever.py
Other components
- Query initiator
- OpenAI embeddings
- Cross-encoder model
Connections 17
- Book PDFs → PDF extraction: reads PDFs
src/retriever.py - PDF extraction → Chunk and embed: passes documents
src/retriever.py - Chunk and embed → OpenAI embeddings: embeds chunks
src/retriever.py - Chunk and embed → Persistent vector store: stores chunks
src/retriever.py - Vector retrieval → Persistent vector store: opens store
src/retriever.py - Vector retrieval → Runtime settings: reads settings
src/retriever.py - Vector retrieval → PDF extraction: loads PDFs
src/retriever.py - Vector retrieval → Chunk and embed: chunks documents
src/retriever.py - Vector retrieval → OpenAI embeddings: configures embeddings
src/retriever.py - Query initiator → Cross-encoder reranker: submits query
- Cross-encoder reranker → Vector retrieval: loads store
src/reranker.py - Cross-encoder reranker → Persistent vector store: searches candidates
src/reranker.py - Cross-encoder reranker → Cross-encoder model: scores pairs
src/reranker.py - Evaluation runner → Golden queries: loads cases
evals/eval_retriever.py - Evaluation runner → Cross-encoder reranker: invokes retrieval
evals/eval_retriever.py - Evaluation runner → Contextual metrics: evaluates results
evals/eval_retriever.py - Evaluation runner → Runtime settings: reads settings
evals/eval_retriever.py