# mrdbourke/simple-local-rag architecture

> Architecture diagram of the GitHub repository mrdbourke/simple-local-rag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mrdbourke/simple-local-rag
- Repository: https://github.com/mrdbourke/simple-local-rag

## Overview

##### Project Type and Purpose
- This is a tutorial project that demonstrates a local retrieval-augmented generation (RAG) pipeline.
- Its primary purpose is to take a large PDF (a 1200-page nutrition textbook) as a data source; preprocess it into smaller text chunks; create embeddings using open-source tools (e.g., Hugging Face sentence transformers, PyTorch); perform similarity search; and then use an LLM (running on a local NVIDIA GPU) to generate answers based on retrieved text.
- The workflow includes elements such as document ingestion, text chunking, embedding creation, search for relevant passages, and LLM-based answer generation—with an optional chat interface as noted in the README.

##### File Structure
- Top-level files include:
- 00-simple-local-rag.ipynb (the main notebook demonstrating the pipeline)
- README.md (which contains the detailed description and flowchart)
- A PDF file (human-nutrition-text.pdf) that serves as the data source
- A requirements.txt (defines dependencies such as torch)
- A folder with video notebooks offering a step-by-step walkthrough.
- The simplicity of the file tree (mostly notebooks and supporting media) indicates that this is an experimental/demo project built primarily for educational purposes rather than a full production architecture, but it still reflects all major processing steps in a RAG pipeline.

##### Key Components
a. Data Source:
- PDF Ingestion: Represent the “human-nutrition-text.pdf” as the main document source.
b. Preprocessing & Chunking:
- Document Parsing and splitting into smaller text chunks (groups of 10 sentences).
c. Embedding Creation:
- Use of embedding models (e.g., sentence transformers or equivalent from Hugging Face) to convert text chunks into numerical vectors (stored as torch.tensors).
- Optional: mention vector databases or indexing if dealing with very large datasets (100k+ embeddings).
d. Query Processing:
- Convert a user query (e.g., “What are the macronutrients?”) into its numerical embedding using the same or a similar embedding model.
e. Retrieval:
- Similarity search to find the most relevant text chunks that closely match the query.
f. LLM Generation:
- Use of a local Large Language Model (LLM; e.g., Gemma or another open-source model) running on the local NVIDIA GPU to generate the final answer based on the retrieved context.
g. Hardware & Environment:
- That all operations (especially computationally heavy tasks like embeddings and LLM inference) are executed on a local NVIDIA GPU (e.g., RTX 4090 with CUDA support).
h. Optional Chat Interface:
- An interface element if a “chat with PDF” feature is implemented.

##### Interactions & Data Flow
- Directional flow starting from the document ingestion (PDF) through each transformation:

##### Architectural Patterns and Design Principles
- Modular design: Break the pipeline into distinct components (ingestion, processing, retrieval, generation, and interface).
- Local Processing Emphasis: Highlight that the entire workflow is designed to run on local hardware (specifically making use of GPU acceleration).
- Efficiency & scalability are considered with the use of numerical embeddings and potential for vector indexing when dealing with large datasets.
- Clear separation of concerns: Data ingestion/processing vs. model execution (LLM) which improves maintainability.

##### Relevant Technologies and Frameworks
- Python (with notebooks as the user interface)
- PyTorch for tensor operations and potentially LLM inference
- Hugging Face libraries for both model and embedding handling
- CUDA (for GPU acceleration on a NVIDIA RTX 4090)
- (Optional) Flash Attention 2 for faster attention computations if compiled and used
- Jupyter Notebook as the interactive development environment

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion Layer
    subgraph "Data Ingestion"
        pdf["PDF Ingestion"]
    end

    %% Preprocessing & Embedding Layer
    subgraph "Preprocessing & Embedding"
        preprocess["Text Preprocessing & Chunking"]
        embed["Embedding Creation"]
    end

    %% Query Processing & Retrieval Layer
    subgraph "Query & Retrieval"
        query["User Query Processing"]
        retrieval["Similarity Search Retrieval"]
        concat["Context Concatenation"]
    end

    %% Answer Generation Layer
    subgraph "Answer Generation"
        llm["LLM Generation"]
    end

    %% Interface Layer
    subgraph "Interface"
        output["Output / Chat Interface"]
    end

    %% Hardware & Environment
    hw["Hardware & Environment Setup"]
    gpu["Local NVIDIA GPU (RTX 4090)"]

    %% Extended Walkthrough (Optional)
    video["Optional Chat Interface / Extended Walkthrough"]

    %% Workflow Diagram Reference (Annotation)
    ref["Workflow Diagram Reference"]

    %% Data Flow Connections
    pdf --> preprocess
    preprocess --> embed
    embed --> query
    query --> retrieval
    retrieval --> concat
    concat --> llm
    llm --> output

    %% Hardware connections
    hw --- gpu
    embed --- gpu
    llm --- gpu

    %% Optional branch for extended walkthrough
    output --> video

    %% Reference annotation connection (for context)
    ref --- pdf

    %% Click Events
    click pdf "https://github.com/mrdbourke/simple-local-rag/blob/main/human-nutrition-text.pdf"
    click preprocess "https://github.com/mrdbourke/simple-local-rag/blob/main/00-simple-local-rag.ipynb"
    click embed "https://github.com/mrdbourke/simple-local-rag/blob/main/00-simple-local-rag.ipynb"
    click query "https://github.com/mrdbourke/simple-local-rag/blob/main/00-simple-local-rag.ipynb"
    click retrieval "https://github.com/mrdbourke/simple-local-rag/blob/main/00-simple-local-rag.ipynb"
    click llm "https://github.com/mrdbourke/simple-local-rag/blob/main/00-simple-local-rag.ipynb"
    click hw "https://github.com/mrdbourke/simple-local-rag/blob/main/requirements.txt"
    click ref "https://github.com/mrdbourke/simple-local-rag/tree/main/images"
    click video "https://github.com/mrdbourke/simple-local-rag/blob/main/video_notebooks/00-simple-local-rag-video.ipynb"

    %% Styling Classes
    classDef data fill:#f9e79f,stroke:#d35400,stroke-width:2px;
    classDef process fill:#aed6f1,stroke:#2471a3,stroke-width:2px;
    classDef query fill:#a9dfbf,stroke:#1e8449,stroke-width:2px;
    classDef model fill:#f5cba7,stroke:#d35400,stroke-width:2px;
    classDef env fill:#d7bde2,stroke:#7d3c98,stroke-width:2px;
    classDef interface fill:#f9e79f,stroke:#d35400,stroke-width:2px;

    %% Assigning Classes
    class pdf data;
    class preprocess,embed process;
    class query,retrieval,concat query;
    class llm model;
    class hw,gpu env;
    class output,video interface;
    class ref env;
```
