How nampham13/rag works
This repository implements a PDF-focused retrieval-augmented generation system, with a Streamlit chat interface and supporting evaluation tools. Its documented document workflow extracts and normalizes PDF content, chunks it, embeds chunks, and stores searchable FAISS indexes with metadata and summaries; queries retrieve and may rerank results before chat generation. The graph also preserves query routing, verification, and evaluation capabilities. Some source was unsampled, so uncertain internal connections are omitted or marked dashed.
Main entry points: rag_pipeline.py RAG pipeline · pipeline_qa.py Retrieval service · LLM_FE.py Chat UI · pdf_loader.py PDF loader
Components 27
Document ingestion
- PDF loader
loaders/pdf_loader.py - Document models
loaders/model/document.py - Block normalization
loaders/model/block.py - Hybrid chunking
chunkers/hybrid_chunker.py
Retrieval pipeline
- RAG pipeline
pipeline/rag_pipeline.py - FAISS vector store
pipeline/vector_store.py - Summary generator
pipeline/summary_generator.py - Vector retriever
pipeline/retriever.py - Retrieval service
pipeline/pipeline_qa.py - Query router
pipeline/query_router.py - Query expander
pipeline/query_expander.py - Chunk verifier
pipeline/chunk_verifier.py
Model providers
- Embedder factory
embedders/embedder_factory.py - Gemma embedder
embedders/providers/ollama/gemma_embedder.py - BGE-M3 embedder
embedders/providers/ollama/bge3_embedder.py - Reranker factory
rerankers/reranker_factory.py - Reranker providers
rerankers/providers/bge_reranker.py
Chat experience
- Chat UI
llm/LLM_FE.py - Chat formatting
llm/chat_handler.py - LLM API client
llm/LLM_API.py
Evaluation
- Evaluation runner
evaluate/evaluator.py - Ground truth
evaluate/model/ground_truth.py - Evaluation metrics
evaluate/metrics/base_metric.py - Evaluation datasets
evaluate/dataset_loader.py
Other components
- User
- PDF documents
- LLM service
Connections 28
- User → Chat UI: asks
llm/LLM_FE.py - Chat UI → Retrieval service: retrieves through
llm/LLM_FE.py - Chat UI → Chat formatting: formats messages
llm/LLM_FE.py - Chat UI → LLM API client: calls
llm/LLM_FE.py - LLM API client → LLM service: requests completion
- PDF documents → PDF loader: loads
README.md - RAG pipeline → PDF loader: loads PDFs
pipeline/rag_pipeline.py - PDF loader → Document models: creates documents
loaders/pdf_loader.py - PDF loader → Block normalization: creates blocks
loaders/pdf_loader.py - RAG pipeline → Hybrid chunking: chunks documents
pipeline/rag_pipeline.py - Hybrid chunking → Document models: consumes documents
chunkers/hybrid_chunker.py - RAG pipeline → Embedder factory: creates embedder
pipeline/rag_pipeline.py - Embedder factory → Gemma embedder: creates
embedders/embedder_factory.py - Embedder factory → BGE-M3 embedder: creates
embedders/embedder_factory.py - RAG pipeline → FAISS vector store: stores vectors
pipeline/rag_pipeline.py - RAG pipeline → Summary generator: generates summaries
pipeline/rag_pipeline.py - RAG pipeline → Vector retriever: uses for search
pipeline/rag_pipeline.py - Retrieval service → RAG pipeline: delegates retrieval
pipeline/pipeline_qa.py - Retrieval service → RAG pipeline: delegates reranking
pipeline/pipeline_qa.py - RAG pipeline → Reranker factory: creates rerankers
pipeline/rag_pipeline.py - Reranker factory → Reranker providers: creates provider
rerankers/reranker_factory.py - Retrieval service → Query router: routes queries
pipeline/pipeline_qa.py - Retrieval service → Query expander: may expand queries
- RAG pipeline → Chunk verifier: uses verification
pipeline/rag_pipeline.py - Evaluation datasets → Evaluation runner: supplies cases
- Ground truth → Evaluation runner: provides relevance labels
evaluate/evaluator.py - Evaluation runner → Evaluation metrics: runs metrics
evaluate/evaluator.py - Evaluation runner → Retrieval service: evaluates answers