How nirdiamant/rag_techniques works
Purpose and Shape
This is an educational RAG-technique repository, not a unified deployable application. It presents 42+ independent, executable notebook tutorials plus Python-script counterparts for selected techniques. Each example is a self-contained pipeline variant rather than a shared product runtime.
Entry Points and Execution
Primary entry points are Jupyter/Colab notebooks in all_rag_techniques/, such as all_rag_techniques/simple_rag.ipynb, and directly runnable Python examples in all_rag_techniques_runnable_scripts/, such as all_rag_techniques_runnable_scripts/simple_rag.py. helper_functions.py is the shared utility boundary available to examples. Local sample corpora are in data/.
Core RAG Baseline
Foundational examples implement the canonical flow: load a document or structured dataset, split/extract retrieval units, embed/index them, retrieve against a query, and provide retrieved context to an LLM for answer generation. Representative paths: all_rag_techniques/simple_rag.ipynb, all_rag_techniques_runnable_scripts/simple_rag.py, data/nike_2023_annual_report.txt.
Index Construction and Context Design
Several examples change the corpus-to-index stage: fixed-size or semantic chunking, proposition extraction, headers, neighboring-context expansion, compression, and generated question augmentation. These establish a boundary between source documents and enriched retrieval records. Representative paths: all_rag_techniques/semantic_chunking.ipynb, all_rag_techniques/document_augmentation.ipynb, all_rag_techniques_runnable_scripts/context_enrichment_window_around_chunk.py.
Query and Retrieval Strategies
Query-time variants rewrite/decompose questions or synthesize hypothetical content, then retrieve via vector, keyword/vector fusion, hierarchical, diversity-aware, or reranked search. These examples are alternatives that can compose conceptually, but are not wired into one common orchestrator. Representative paths: all_rag_techniques/query_transformations.ipynb, all_rag_techniques/fusion_retrieval.ipynb, all_rag_techniques/reranking.ipynb.
Adaptive and Agentic Control Flows
Iterative examples introduce decision loops around retrieval: classify/adapt strategy, grade relevance, refine knowledge/query, retry retrieval, and assess generated answers. Architecturally, these make retrieval conditional rather than a single pass. Representative paths: all_rag_techniques/self_rag.ipynb, all_rag_techniques/crag.ipynb, all_rag_techniques_runnable_scripts/adaptive_retrieval.py.
Structured, Hierarchical, and Multimodal Retrieval
Advanced architectures replace or augment flat vector retrieval with knowledge graphs, graph traversal, recursive summary trees, memory stores, or image-based/captioned representations. Graph approaches explicitly combine entity/relation retrieval with source passages; local attribution retains sentence-level provenance. Representative paths: all_rag_techniques/graph_rag.ipynb, all_rag_techniques/Microsoft_GraphRag.ipynb, all_rag_techniques/graph_rag_local_attribution.ipynb.
Evaluation and External Runtime Boundaries
Evaluation tutorials assess answer correctness, faithfulness, contextual relevance, hallucinations, and citations, using local Q&A datasets where applicable. The repository demonstrates integrations rather than pinning a single stack: LangChain and LlamaIndex examples; OpenAI/GPT-4, Ollama, Hugging Face/FAISS, Contextual AI, Milvus, NetworkX, DeepEval, GroUSE, RAGAS, and Open-RAG-Eval appear as technique-specific external services/libraries. Representative paths: evaluation/end-2-end_rag_evaluation.ipynb, evaluation/evalute_rag.py, data/q_a.json.
Main entry points: simple_rag.ipynb Load and split · simple_rag.py Embed and index · adaptive_retrieval.py Adaptive retrieval loop · helper_functions.py Shared helpers
Components 18
Independent tutorial examples
Notebook and script variants are executed independently.
- Notebook tutorials · 42+ self-contained RAG technique tutorials.
all_rag_techniques/simple_rag.ipynb - Runnable scripts · Script counterparts for selected techniques.
all_rag_techniques_runnable_scripts/simple_rag.py - Shared helpers · Optional shared support used by examples.
helper_functions.py
Canonical RAG flow
A conceptual baseline shared across introductory examples.
- Local documents
data/nike_2023_annual_report.txt - Load and split · Extracts chunks or structured retrieval units.
all_rag_techniques/simple_rag.ipynb - Embed and index
all_rag_techniques_runnable_scripts/simple_rag.py - Retrieve context
- Generate answer · Supplies retrieved context to an LLM.
Advanced retrieval patterns
- Context enrichment · Chunking, propositions, headers, windows, compression, and question augmentation.
all_rag_techniques/semantic_chunking.ipynb - Document augmentation · Generates enriched retrieval records from source content.
all_rag_techniques/document_augmentation.ipynb - Query transformations · Rewrites, decomposes, or synthesizes hypothetical queries.
all_rag_techniques/query_transformations.ipynb - Fusion retrieval · Combines keyword and vector retrieval.
all_rag_techniques/fusion_retrieval.ipynb - Reranking
all_rag_techniques/reranking.ipynb - Adaptive retrieval loop · Grades, refines, retries, and assesses answers conditionally.
all_rag_techniques_runnable_scripts/adaptive_retrieval.py - Graph RAG · Combines entity/relation traversal with source passages.
all_rag_techniques/graph_rag.ipynb
Evaluation
- Q&A datasets
data/q_a.json - RAG evaluation · Measures correctness, faithfulness, relevance, hallucinations, and citations.
evaluation/end-2-end_rag_evaluation.ipynb
Other components
- Technique-specific stack · Examples select their own model, framework, vector store, and evaluator integrations.
Connections 18
- Notebook tutorials → Shared helpers: may use
- Runnable scripts → Shared helpers: may use
- Local documents → Load and split: load
- Load and split → Embed and index: embed units
- Embed and index → Retrieve context: search
- Retrieve context → Generate answer: context
- Context enrichment → Load and split: changes unit design
- Document augmentation → Embed and index: enriches records
- Query transformations → Retrieve context: changes query
- Fusion retrieval → Retrieve context: alternative search
- Reranking → Retrieve context: reorders results
- Adaptive retrieval loop → Retrieve context: conditional retries
- Adaptive retrieval loop → Generate answer: assesses output
- Graph RAG → Retrieve context: augments retrieval
- Q&A datasets → RAG evaluation: benchmark inputs
- Generate answer → RAG evaluation: answers assessed
- Notebook tutorials → Technique-specific stack: integrate
- Runnable scripts → Technique-specific stack: integrate