# nirdiamant/rag_techniques architecture

> Architecture diagram of the GitHub repository nirdiamant/rag_techniques, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nirdiamant/rag_techniques
- Repository: https://github.com/nirdiamant/rag_techniques

## Overview

#### Purpose and Shape
This is an educational RAG-technique repository, not a unified deployable application. It presents 42+ independent, executable notebook tutorials plus Python-script counterparts for selected techniques. Each example is a self-contained pipeline variant rather than a shared product runtime.

#### Entry Points and Execution
Primary entry points are Jupyter/Colab notebooks in `all_rag_techniques/`, such as `all_rag_techniques/simple_rag.ipynb`, and directly runnable Python examples in `all_rag_techniques_runnable_scripts/`, such as `all_rag_techniques_runnable_scripts/simple_rag.py`. `helper_functions.py` is the shared utility boundary available to examples. Local sample corpora are in `data/`.

#### Core RAG Baseline
Foundational examples implement the canonical flow: load a document or structured dataset, split/extract retrieval units, embed/index them, retrieve against a query, and provide retrieved context to an LLM for answer generation. Representative paths: `all_rag_techniques/simple_rag.ipynb`, `all_rag_techniques_runnable_scripts/simple_rag.py`, `data/nike_2023_annual_report.txt`.

#### Index Construction and Context Design
Several examples change the corpus-to-index stage: fixed-size or semantic chunking, proposition extraction, headers, neighboring-context expansion, compression, and generated question augmentation. These establish a boundary between source documents and enriched retrieval records. Representative paths: `all_rag_techniques/semantic_chunking.ipynb`, `all_rag_techniques/document_augmentation.ipynb`, `all_rag_techniques_runnable_scripts/context_enrichment_window_around_chunk.py`.

#### Query and Retrieval Strategies
Query-time variants rewrite/decompose questions or synthesize hypothetical content, then retrieve via vector, keyword/vector fusion, hierarchical, diversity-aware, or reranked search. These examples are alternatives that can compose conceptually, but are not wired into one common orchestrator. Representative paths: `all_rag_techniques/query_transformations.ipynb`, `all_rag_techniques/fusion_retrieval.ipynb`, `all_rag_techniques/reranking.ipynb`.

#### Adaptive and Agentic Control Flows
Iterative examples introduce decision loops around retrieval: classify/adapt strategy, grade relevance, refine knowledge/query, retry retrieval, and assess generated answers. Architecturally, these make retrieval conditional rather than a single pass. Representative paths: `all_rag_techniques/self_rag.ipynb`, `all_rag_techniques/crag.ipynb`, `all_rag_techniques_runnable_scripts/adaptive_retrieval.py`.

#### Structured, Hierarchical, and Multimodal Retrieval
Advanced architectures replace or augment flat vector retrieval with knowledge graphs, graph traversal, recursive summary trees, memory stores, or image-based/captioned representations. Graph approaches explicitly combine entity/relation retrieval with source passages; local attribution retains sentence-level provenance. Representative paths: `all_rag_techniques/graph_rag.ipynb`, `all_rag_techniques/Microsoft_GraphRag.ipynb`, `all_rag_techniques/graph_rag_local_attribution.ipynb`.

#### Evaluation and External Runtime Boundaries
Evaluation tutorials assess answer correctness, faithfulness, contextual relevance, hallucinations, and citations, using local Q&A datasets where applicable. The repository demonstrates integrations rather than pinning a single stack: LangChain and LlamaIndex examples; OpenAI/GPT-4, Ollama, Hugging Face/FAISS, Contextual AI, Milvus, NetworkX, DeepEval, GroUSE, RAGAS, and Open-RAG-Eval appear as technique-specific external services/libraries. Representative paths: `evaluation/end-2-end_rag_evaluation.ipynb`, `evaluation/evalute_rag.py`, `data/q_a.json`.

Main entry points: [`all_rag_techniques/simple_rag.ipynb`](https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques/simple_rag.ipynb) (Load and split), [`all_rag_techniques_runnable_scripts/simple_rag.py`](https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques_runnable_scripts/simple_rag.py) (Embed and index), [`all_rag_techniques_runnable_scripts/adaptive_retrieval.py`](https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques_runnable_scripts/adaptive_retrieval.py) (Adaptive retrieval loop), [`helper_functions.py`](https://github.com/nirdiamant/rag_techniques/blob/main/helper_functions.py) (Shared helpers)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_examples["Independent tutorial examples"]
  node_notebooks["Notebook tutorials<br/>Jupyter examples<br/>[simple_rag.ipynb]"]
  node_scripts["Runnable scripts<br/>Python examples<br/>[simple_rag.py]"]
  node_helpers["Shared helpers<br/>utility boundary"]
end

subgraph group_baseline["Canonical RAG flow"]
  node_corpus["Local documents<br/>sample corpus"]
  node_ingest["Load and split<br/>retrieval units<br/>[simple_rag.ipynb]"]
  node_index[("Embed and index<br/>vector index<br/>[simple_rag.py]")]
  node_retrieve["Retrieve context<br/>search stage"]
  node_generate["Generate answer<br/>LLM generation"]
end

subgraph group_advanced["Advanced retrieval patterns"]
  node_context_design["Context enrichment<br/>index variant"]
  node_document_augmentation["Document augmentation<br/>index variant"]
  node_query_strategies["Query transformations<br/>query variant"]
  node_fusion["Fusion retrieval<br/>retrieval variant"]
  node_reranking["Reranking<br/>retrieval variant<br/>[reranking.ipynb]"]
  node_adaptive{{"Adaptive retrieval loop<br/>agentic control flow"}}
  node_graph_retrieval["Graph RAG<br/>structured retrieval<br/>[graph_rag.ipynb]"]
end

subgraph group_evaluation["Evaluation"]
  node_evaluation_data["Q&amp;A datasets<br/>evaluation data<br/>[q_a.json]"]
  node_evaluation["RAG evaluation<br/>evaluation tutorial"]
end

node_external_stack{{"Technique-specific stack<br/>external integrations"}}

node_notebooks -.->|"may use"| node_helpers
node_scripts -.->|"may use"| node_helpers
node_corpus -->|"load"| node_ingest
node_ingest -->|"embed units"| node_index
node_index -->|"search"| node_retrieve
node_retrieve -->|"context"| node_generate
node_context_design -.->|"changes unit design"| node_ingest
node_document_augmentation -.->|"enriches records"| node_index
node_query_strategies -.->|"changes query"| node_retrieve
node_fusion -.->|"alternative search"| node_retrieve
node_reranking -.->|"reorders results"| node_retrieve
node_adaptive -.->|"conditional retries"| node_retrieve
node_adaptive -.->|"assesses output"| node_generate
node_graph_retrieval -.->|"augments retrieval"| node_retrieve
node_evaluation_data -->|"benchmark inputs"| node_evaluation
node_generate -.->|"answers assessed"| node_evaluation
node_notebooks -.->|"integrate"| node_external_stack
node_scripts -.->|"integrate"| node_external_stack

click node_notebooks "https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques/simple_rag.ipynb"
click node_scripts "https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques_runnable_scripts/simple_rag.py"
click node_helpers "https://github.com/nirdiamant/rag_techniques/blob/main/helper_functions.py"
click node_corpus "https://github.com/nirdiamant/rag_techniques/blob/main/data/nike_2023_annual_report.txt"
click node_ingest "https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques/simple_rag.ipynb"
click node_index "https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques_runnable_scripts/simple_rag.py"
click node_context_design "https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques/semantic_chunking.ipynb"
click node_document_augmentation "https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques/document_augmentation.ipynb"
click node_query_strategies "https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques/query_transformations.ipynb"
click node_fusion "https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques/fusion_retrieval.ipynb"
click node_reranking "https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques/reranking.ipynb"
click node_adaptive "https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques_runnable_scripts/adaptive_retrieval.py"
click node_graph_retrieval "https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques/graph_rag.ipynb"
click node_evaluation_data "https://github.com/nirdiamant/rag_techniques/blob/main/data/q_a.json"
click node_evaluation "https://github.com/nirdiamant/rag_techniques/blob/main/evaluation/end-2-end_rag_evaluation.ipynb"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_notebooks,node_scripts,node_helpers toneBlue
class node_corpus,node_ingest,node_index,node_retrieve,node_generate toneAmber
class node_context_design,node_document_augmentation,node_query_strategies,node_fusion,node_reranking,node_adaptive,node_graph_retrieval toneMint
class node_evaluation_data,node_evaluation toneRose
class node_external_stack toneNeutral
```

## Components

### Independent tutorial examples

Notebook and script variants are executed independently.

- [Notebook tutorials](https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques/simple_rag.ipynb): Jupyter examples · 42+ self-contained RAG technique tutorials. · `all_rag_techniques/simple_rag.ipynb`
- [Runnable scripts](https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques_runnable_scripts/simple_rag.py): Python examples · Script counterparts for selected techniques. · `all_rag_techniques_runnable_scripts/simple_rag.py`
- [Shared helpers](https://github.com/nirdiamant/rag_techniques/blob/main/helper_functions.py): utility boundary · Optional shared support used by examples. · `helper_functions.py`

### Canonical RAG flow

A conceptual baseline shared across introductory examples.

- [Local documents](https://github.com/nirdiamant/rag_techniques/blob/main/data/nike_2023_annual_report.txt): sample corpus · `data/nike_2023_annual_report.txt`
- [Load and split](https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques/simple_rag.ipynb): retrieval units · Extracts chunks or structured retrieval units. · `all_rag_techniques/simple_rag.ipynb`
- [Embed and index](https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques_runnable_scripts/simple_rag.py): vector index · `all_rag_techniques_runnable_scripts/simple_rag.py`
- **Retrieve context**: search stage
- **Generate answer**: LLM generation · Supplies retrieved context to an LLM.

### Advanced retrieval patterns

- [Context enrichment](https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques/semantic_chunking.ipynb): index variant · Chunking, propositions, headers, windows, compression, and question augmentation. · `all_rag_techniques/semantic_chunking.ipynb`
- [Document augmentation](https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques/document_augmentation.ipynb): index variant · Generates enriched retrieval records from source content. · `all_rag_techniques/document_augmentation.ipynb`
- [Query transformations](https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques/query_transformations.ipynb): query variant · Rewrites, decomposes, or synthesizes hypothetical queries. · `all_rag_techniques/query_transformations.ipynb`
- [Fusion retrieval](https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques/fusion_retrieval.ipynb): retrieval variant · Combines keyword and vector retrieval. · `all_rag_techniques/fusion_retrieval.ipynb`
- [Reranking](https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques/reranking.ipynb): retrieval variant · `all_rag_techniques/reranking.ipynb`
- [Adaptive retrieval loop](https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques_runnable_scripts/adaptive_retrieval.py): agentic control flow · Grades, refines, retries, and assesses answers conditionally. · `all_rag_techniques_runnable_scripts/adaptive_retrieval.py`
- [Graph RAG](https://github.com/nirdiamant/rag_techniques/blob/main/all_rag_techniques/graph_rag.ipynb): structured retrieval · Combines entity/relation traversal with source passages. · `all_rag_techniques/graph_rag.ipynb`

### Evaluation

- [Q&A datasets](https://github.com/nirdiamant/rag_techniques/blob/main/data/q_a.json): evaluation data · `data/q_a.json`
- [RAG evaluation](https://github.com/nirdiamant/rag_techniques/blob/main/evaluation/end-2-end_rag_evaluation.ipynb): evaluation tutorial · Measures correctness, faithfulness, relevance, hallucinations, and citations. · `evaluation/end-2-end_rag_evaluation.ipynb`

### Other components

- **Technique-specific stack**: external integrations · Examples select their own model, framework, vector store, and evaluator integrations.

## Connections

- Notebook tutorials → Shared helpers: may use
- Runnable scripts → Shared helpers: may use
- Local documents → Load and split: load
- Load and split → Embed and index: embed units
- Embed and index → Retrieve context: search
- Retrieve context → Generate answer: context
- Context enrichment → Load and split: changes unit design
- Document augmentation → Embed and index: enriches records
- Query transformations → Retrieve context: changes query
- Fusion retrieval → Retrieve context: alternative search
- Reranking → Retrieve context: reorders results
- Adaptive retrieval loop → Retrieve context: conditional retries
- Adaptive retrieval loop → Generate answer: assesses output
- Graph RAG → Retrieve context: augments retrieval
- Q&A datasets → RAG evaluation: benchmark inputs
- Generate answer → RAG evaluation: answers assessed
- Notebook tutorials → Technique-specific stack: integrate
- Runnable scripts → Technique-specific stack: integrate
