# faisalshahbaz/rag_techniques architecture

> Architecture diagram of the GitHub repository faisalshahbaz/rag_techniques, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/faisalshahbaz/rag_techniques
- Repository: https://github.com/faisalshahbaz/rag_techniques

## Overview

The project is a repository of advanced Retrieval-Augmented Generation (RAG) techniques. Its purpose is to serve both as a knowledge hub and a hands-on playground for experimenting with different retrieval strategies for combining information retrieval and generative AI. This repository is highly modular, with each Jupyter Notebook in the “all_rag_techniques” folder representing a distinct technique ranging from simple RAG implementations to more advanced methods like adaptive retrieval and even a controllable RAG agent.

##### Main Components
- Techniques Module: Represent the “all_rag_techniques” directory as the core component where each notebook (such as “simple_rag.ipynb”, “context_enrichment_window_around_chunk.ipynb”, etc.) acts as an independent module that implements a specific RAG enhancement.
- Evaluation Module: Include the “evaluation” folder with its notebooks and scripts that define and perform model evaluations.
- Data Storage: Represent components for “data” (e.g., the PDF and JSON file) that are used for testing and demonstrating retrieval methods.
- Helper Functions & Configuration: The “helper_functions.py” file may be indicated as a shared utility library that supports multiple techniques.
- External Dependencies/Requirements: The “requirements.txt” file implies dependencies such as Python libraries and frameworks that support the system’s operations.

##### Relationships and Interactions
- Modular Techniques: Specify that the different RAG techniques interact with a common retrieval engine. For example, show how “adaptive_retrieval”, “self_rag”, and “crag” all feed into a central pipeline.
- Evaluation Feedback: Illustrate that the evaluation components (from the “evaluation” folder) feed results back into the system, enabling adjustments (e.g., in adaptive methods or feedback loops) for optimizing retrieval accuracy.
- External Interfaces: Include any external service components like language models (LLMs), vector databases, or even knowledge graphs, which may be integrated in techniques such as “graph_rag” or “hypothetical document embedding”.

##### Architectural Patterns and Design Principles
- Modularity: Emphasize how each notebook represents a distinct, testable module that can be integrated or used independently. This modularity supports experimentation and component interchangeability.
- Dynamic and Adaptive Processing: Clearly indicate the existence of adaptive feedback loops (e.g., in “retrieval_with_feedback_loop” and “self_rag”) designed to refine system performance.
- Layered Architecture: Consider a layered approach where you separate user input, query processing, retrieval logic, and evaluation into distinct layers.
- Extensibility: The design should show clear extension points for new retrieval techniques and evaluation metrics, aligning with the community-driven, open-source nature of the project.

##### Relevant Technologies and Concepts
- Python & Jupyter: The primary implementation medium, noted by the notebooks and Python files.
- NLP & LLMs: Underlying technologies that drive query transformation, retrieval reranking, and response generation.
- Evaluation Metrics & Experimentation: A dedicated evaluation layer that uses notebooks and scripts for performance assessment.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interaction Layer
    subgraph "User Interaction Layer"
        A["User Input Query"]:::input
    end

    %% Processing Layer
    subgraph "Processing Layer"
        B["Query Transformation"]:::process
        C["Retrieval Engine"]:::process
        D["Techniques Module"]:::process
        E["Response Generation"]:::process
        F["Adaptive Feedback Loop"]:::process
    end

    %% Data and External Layer
    subgraph "Data and External Layer"
        DS["Data Storage"]:::storage
        EX["External Dependencies"]:::external
        HF["Helper Functions"]:::external
    end

    %% Evaluation Module (placed outside subgraphs to emphasize its role)
    Eval["Evaluation Module"]:::evaluation

    %% Data Flow
    A -->|"processes"| B
    B -->|"transforms"| C
    C -->|"routes"| D
    D -->|"executes"| E
    E -->|"sendsResults"| Eval
    Eval -->|"feedback"| C
    C -->|"retrieves"| DS
    C -->|"calls"| EX
    HF -->|"supports"| C
    HF -->|"supports"| D
    E -->|"triggers"| F
    F -->|"feedsBack"| C

    %% Click Events
    click D "https://github.com/faisalshahbaz/rag_techniques/tree/main/all_rag_techniques/"
    click Eval "https://github.com/faisalshahbaz/rag_techniques/tree/main/evaluation/"
    click DS "https://github.com/faisalshahbaz/rag_techniques/tree/main/data/"
    click HF "https://github.com/faisalshahbaz/rag_techniques/blob/main/helper_functions.py"
    click EX "https://github.com/faisalshahbaz/rag_techniques/blob/main/requirements.txt"

    %% Styles
    classDef input fill:#ccf,stroke:#333,stroke-width:2px;
    classDef process fill:#fcf,stroke:#333,stroke-width:2px;
    classDef storage fill:#cfc,stroke:#333,stroke-width:2px;
    classDef external fill:#fcc,stroke:#333,stroke-width:2px;
    classDef evaluation fill:#ffc,stroke:#333,stroke-width:2px;
```
