How peishuen/rag-personal-document-assistant works
This is a Streamlit personal-document RAG assistant: users upload documents, ask questions, and receive answers grounded in retrieved passages with page-level citations. The main flow covers ingestion, adaptive chunking, embedding and storage, hybrid retrieval, answer generation, grounding checks, and citation. The README also documents batch and cross-document Q&A, document management, and retrieval evaluation. Some subsystems are documented but not sampled; their inclusion and links rely on the README, with uncertain wiring omitted.
Main entry points: app.py Streamlit app · chroma_store.py Chroma store · batch_loader.py Batch loader · hybrid_ranker.py Hybrid ranker
Components 23
Assistant UI
- Streamlit app
app.py
Ingestion and indexing
- Batch loader
src/ingestion/batch_loader.py - PDF loader
src/ingestion/pdf_loader.py - Text loader
src/ingestion/text_loader.py - Scanned PDF OCR
src/ingestion/ocr_loader.py - Adaptive chunker
src/chunking/chunker.py - Embedding service
src/embeddings/embedder.py - Chroma store
src/vectorstore/chroma_store.py - Persistent chunk index
data/chroma_db
Retrieval
- Cross-document search
src/retrieval/cross_document.py - Hybrid ranker
src/retrieval/hybrid_ranker.py - BM25 search
src/retrieval/bm25_retriever.py - Semantic search
src/retrieval/semantic_retriever.py - Query rewriter
src/generation/query_rewriter.py
Answer and citations
- Answer generator
src/generation/generator.py - Grounding check
src/generation/grounding_check.py - Citation tracker
src/citation/citation_tracker.py
Evaluation
- Retrieval evaluation
evaluation/evaluation.py - Embedding comparison
evaluation/embedding_comparison.py
Other components
- Document user
- DashScope API
- Qwen language model
- Tesseract OCR
Connections 24
- Document user → Streamlit app: uploads and asks
- Streamlit app → Batch loader: loads documents
- Batch loader → PDF loader: routes PDFs
- Batch loader → Text loader: routes text
- Batch loader → Scanned PDF OCR: routes scans
- Scanned PDF OCR → Tesseract OCR: extracts text
- Streamlit app → Adaptive chunker: chunks pages
- Streamlit app → Embedding service: embeds chunks
- Embedding service → DashScope API: requests vectors
- Streamlit app → Chroma store: stores chunks
- Chroma store → Persistent chunk index: writes and reads
- Streamlit app → Chroma store: manages sources
- Streamlit app → Query rewriter: rewrites follow-ups
- Streamlit app → Cross-document search: searches documents
- Cross-document search → Hybrid ranker: combines retrieval
- Hybrid ranker → BM25 search: requests keyword results
- Hybrid ranker → Semantic search: requests semantic results
- BM25 search → Chroma store: reads chunks
- Semantic search → Embedding service: embeds query
- Semantic search → Chroma store: searches vectors
- Streamlit app → Answer generator: generates answer
- Answer generator → Qwen language model: requests completion
- Streamlit app → Grounding check: checks support
- Streamlit app → Citation tracker: formats sources