# promtengineer/localgpt architecture

> Architecture diagram of the GitHub repository promtengineer/localgpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/promtengineer/localgpt
- Repository: https://github.com/promtengineer/localgpt

## Overview

##### Project Type & Purpose
- Project Type: Open-source, local RAG (Retrieval-Augmented Generation) tool with both CLI and web UI layers.
- Purpose: Ingest arbitrary documents, build a local vector store, run a private LLM-based Q&A pipeline without sending data off-premise, and expose both CLI and REST/UI interfaces.

##### File Structure
Top-level folders & files reveal these logical modules:
- SOURCE_DOCUMENTS/ ← raw inputs for ingestion
- ingest.py ← pipeline to load documents, embed them, store them in Chroma
- run_localGPT.py ← CLI tool that handles user queries via local model
- run_localGPT_API.py ← Python service (Werkzeug/Flask) exposing REST endpoints
- localGPTUI/ ← front-end web application (templates, static assets, localGPTUI.py)
- gaudi_utils/ ← hardware-specific utilities (HPU support)
- constants.py, prompt_template_utils.py, utils.py ← shared config, code templates, helpers
- Dockerfile, Dockerfile_hpu ← containerization for GPU/HPU/MPS setups

##### Key Components
a. Data Ingestion Pipeline
- Document Loader (LangChain loaders via constants.py)
- Embedding Model (InstructorEmbeddings)
- Vector Store (ChromaDB → local DB folder)
b. Inference Pipeline
- Retrieval (similarity search from Chroma)
- LLM Model Loader (HuggingFace or llama-cpp via load_models.py)
- Generation Engine (run_localGPT.py or run_localGPT_API.py logic)
c. Interfaces
- CLI Interface (run_localGPT.py)
- REST API Service (run_localGPT_API.py)
- Web UI (localGPTUI.py + HTML/CSS/JS under localGPTUI/static & templates)
d. External Dependencies / Infrastructure
- HuggingFace model hub or local GGML/GGUF artifacts
- LangChain library
- GPU/CPU/HPU/MPS runtimes
- Docker container (optional)

##### Relationships & Data Flow
- Source Documents → (ingest.py) → Embedding Model → Vector Store
- User Query (CLI or UI) → API or direct CLI script
- API/CLI → Retrieval from Vector Store → Context Chunks
- Context + Query → LLM Inference → Answer
- Answer → CLI stdout or API JSON → UI renderer

##### Architectural Patterns & Principles
- Layered Architecture: Data layer, Storage layer, Compute layer, Presentation layer
- Pipeline Pattern: ingestion → storage → retrieval → generation
- Adapter/Strategy Pattern: plugging in different loader models or hardware backends via gaudi_utils
- Microservices (lightweight): separate CLI, API service, and UI processes

##### Technologies & Frameworks to Call Out
- Python 3.10+, LangChain, ChromaDB
- HuggingFace Transformers / llama-cpp-python
- InstructorEmbeddings for vectorization
- Flask/Werkzeug for API (embedded in run_localGPT_API.py)
- Jinja2 templates + Bootstrap/jQuery for the web UI
- Docker / Dockerfile_hpu for containerized GPU or HPU execution

##### Important: Be Very Detailed
- Two UI options if desired: CLI vs. streamlit/web UI.
- Don’t over-simplify—capture every major Python script as its own component.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        SD["SOURCE_DOCUMENTS"]:::storage
        SD -->|load docs| INGEST["ingest.py"]:::compute
        INGEST -->|uses registry| CONST["constants.py"]:::compute
        INGEST -->|uses helpers| UTILS["utils.py"]:::compute
    end

    %% Compute Layer
    subgraph "Compute Layer"
        EMB_ADAPT["gaudi_utils/embeddings.py"]:::compute
        EMB_PIPE["gaudi_utils/pipeline.py"]:::compute
        EMB_ADAPT --> EMB_PIPE
        INGEST -->|embed documents| EMB_PIPE
        EMB_PIPE -->|store embeddings| CHROMA["ChromaDB (local folder)"]:::storage

        RETRIEVE["Retrieval"]:::compute
        CHROMA -->|similarity search| RETRIEVE

        subgraph "LLM Inference"
            MODELS["load_models.py"]:::compute
            GAUDI_PIPE["gaudi_utils/pipeline.py"]:::compute
            RETRIEVE -->|context chunks| MODELS
            MODELS -->|select hardware| GAUDI_PIPE
            GAUDI_PIPE -->|generate answer| GEN["Generation Engine"]:::compute
        end
    end

    %% Presentation Layer
    subgraph "Presentation Layer"
        CLI["run_localGPT.py"]:::interface
        API["run_localGPT_API.py"]:::interface
        UI_APP["localGPTUI/localGPTUI.py"]:::interface
        CLI -->|query| RETRIEVE
        API -->|POST /chat| RETRIEVE
        UI_APP -->|AJAX| API
        API -->|JSON answer| UI_APP
    end

    %% Web UI Details
    subgraph "Web UI Assets"
        TEMPL["templates/home.html"]:::interface
        STATIC["static/"]:::interface
        UI_APP --> TEMPL
        UI_APP --> STATIC
    end

    %% Prompt & Config
    subgraph "Utilities"
        PROMPT["prompt_template_utils.py"]:::compute
        CONST --> PROMPT
        UTILS --> PROMPT
    end

    %% Infrastructure & External
    subgraph "Infrastructure"
        DOCKER_GPU["Dockerfile"]:::infra
        DOCKER_HPU["Dockerfile_hpu"]:::infra
        HF["HuggingFace Hub / GGML"]:::infra
        LANGCHAIN["LangChain Library"]:::infra
        RUNTIME["GPU/CPU/HPU/MPS"]:::infra
        MODELS --> HF
        INGEST --> LANGCHAIN
        MODELS --> RUNTIME
    end

    %% Connections to Generation
    GEN -->|answer| CLI
    GEN -->|answer JSON| API

    %% Click Events
    click SD "https://github.com/promtengineer/localgpt/tree/main/SOURCE_DOCUMENTS/"
    click INGEST "https://github.com/promtengineer/localgpt/blob/main/ingest.py"
    click CONST "https://github.com/promtengineer/localgpt/blob/main/constants.py"
    click UTILS "https://github.com/promtengineer/localgpt/blob/main/utils.py"
    click EMB_ADAPT "https://github.com/promtengineer/localgpt/blob/main/gaudi_utils/embeddings.py"
    click EMB_PIPE "https://github.com/promtengineer/localgpt/blob/main/gaudi_utils/pipeline.py"
    click CLI "https://github.com/promtengineer/localgpt/blob/main/run_localGPT.py"
    click API "https://github.com/promtengineer/localgpt/blob/main/run_localGPT_API.py"
    click UI_APP "https://github.com/promtengineer/localgpt/blob/main/localGPTUI/localGPTUI.py"
    click TEMPL "https://github.com/promtengineer/localgpt/blob/main/localGPTUI/templates/home.html"
    click STATIC "https://github.com/promtengineer/localgpt/tree/main/localGPTUI/static/"
    click PROMPT "https://github.com/promtengineer/localgpt/blob/main/prompt_template_utils.py"
    click DOCKER_GPU "https://github.com/promtengineer/localgpt/tree/main/Dockerfile"
    click DOCKER_HPU "https://github.com/promtengineer/localgpt/tree/main/Dockerfile_hpu"

    %% Styles
    classDef storage fill:#9f6,stroke:#060;
    classDef compute fill:#6cf,stroke:#036;
    classDef interface fill:#f96,stroke:#f60;
    classDef infra fill:#ccc,stroke:#666;
```
