How pdichone/advanced-rag-techniques works
Project type and purpose
• This is an open-source AI tool or library module (part of “The AI Guild”) focused on retrieval-augmented QA and document processing.
• The README positions it as a code module in a community for building AI solutions—no frontend or UI, just backend Python scripts and data.
File structure
Top-level files and directories:
• data/ (holds raw documents; in this case, a PDF)
• dpr_technique.py (Dense Passage Retrieval setup)
• expansion_queries.py & expansion_answer.py (query & answer expansion logic)
• reranking.py (reordering retrieved passages)
• helper_utils.py (shared utilities for preprocessing, I/O, etc.)
• requirements.txt (pins dependencies)
• CLAUDE.md (notes about an LLM, likely an external service)
Patterns:
• Each major step in the retrieval-augmented QA pipeline is encapsulated in its own module.
• There is a clear separation of concerns: data ingestion, query expansion, retrieval (DPR), reranking, and answer expansion.
Key README insights
• External dependencies implied: deep-learning framework (e.g., PyTorch/Transformers), vector store or embeddings service, and an LLM API (e.g., Claude).
Main components
a. Data Ingestion & Preprocessing
• Component: helper_utils.py + data/
• Role: load PDF, clean text, chunk and index documents.
b. Embedding & Retrieval
• Component: dpr_technique.py
• Role: generate dense embeddings, query the vector store.
c. Query Expansion
• Component: expansion_queries.py
• Role: reformulate user queries for better recall.
d. Reranking
• Component: reranking.py
• Role: use a cross-encoder or LLM to reorder top-k passages.
e. Answer Expansion
• Component: expansion_answer.py
• Role: generate final answer from reranked passages via LLM.
f. External Services
• Vector database or in-memory FAISS index
• LLM API (e.g., Anthropic Claude, OpenAI)
Relationships & data flow
• User Query → Query Expansion → DPR Retrieval → Top-n Passages → Reranking → Best Passages → Answer Expansion → Final Response
• Data preprocessing and indexing happen upstream and feed the vector store.
Architectural patterns & principles
• Pipeline pattern: each script is a stage in a linear data flow.
• Separation of concerns: modular, single-purpose components.
• Extensibility points: swap out DPR with BM25, plug in different LLMs via helper_utils.