# getumbrel/llama-gpt architecture

> Architecture diagram of the GitHub repository getumbrel/llama-gpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/getumbrel/llama-gpt
- Repository: https://github.com/getumbrel/llama-gpt

## Overview

##### Project type and purpose
- Full-stack application comprised of:
– A React/Next.js front-end (“ui” folder)
– A Python-based, OpenAI-compatible API back-end (“api” folder, llama-cpp-python)
– Optional GPU acceleration (CUDA Dockerfiles)
– Model storage (local volume)
– Helm/Kubernetes and Docker Compose deployment manifests

##### File structure
- Top-level directories:
– ui/ : Next.js app, pages, components, hooks, styles, Dockerfile, k8s manifest
– api/ : Shell script to launch the Python API server wrapping llama-cpp
– cuda/ : Dockerfiles and scripts for NVIDIA GPU support
– deploy/kubernetes/ : Kustomize manifests for UI & API Deployments and Services
– models/ : Volume mount point for downloaded model files
- Root scripts: run.sh, run-mac.sh orchestrate Docker Compose setups
- Docker Compose files: cpu (GGML), cuda, mac, etc.

##### Key insights from README
- The system exposes two HTTP services:
– UI on port 3000 (Next.js SSR + React SPA)
– API on port 3001 (OpenAI-compatible endpoints)
- Models auto-download into /models on first run
- Supports CPU (GGML) or GPU (CUDA) inference
- Kubernetes manifests allow cloud or on-prem deployment

##### Main components
a. User’s Browser
– React UI (Next.js server)
– Local state & storage (IndexedDB/localStorage for conversation history)
b. UI Service Container
– Next.js server (pages/api proxies, SSR)
– Static assets, i18n, Tailwind CSS
c. API Service Container
– Python Flask/FastAPI (llama-cpp-python) exposing /v1/chat/completions etc.
– Drops in as OpenAI replacement
d. Model Storage Volume
– Shared Docker/K8s volume at /models
– Contains GGML or GGUF quantized weights
e. Inference Engine
– llama.cpp library (CPU) or CUDA-enabled backend
f. External Services (optional)
– Google Search plugin API (pages/api/google.ts)
– (Future) external plugins
g. Orchestration & Deployment
– Docker Compose network connecting ui & api containers
– Kubernetes Deployments + Services under namespace “llama”

##### Component interactions & data flow
- Browser → UI Service (HTTP GET/POST)
- UI Service → API Service (HTTP REST/OpenAI API calls)
- API Service → llama.cpp (in-process library call)
- llama.cpp → Model files on volume → returns tokens
- API Service → UI Service → Browser (streamed Completions)
- UI Service → Local browser storage (save conversations)
- Optional: UI Service → Google Search Service → UI Service

##### Architectural patterns & design principles
- Separation of concerns: UI vs. inference API
- Microservice-style deployment (2 containers, shared volume)
- Swap-in inference backends (CPU/GPU builds) via Dockerfiles
- Declarative infra: Docker Compose & Kubernetes Kustomize
- API compatibility: mimic OpenAI endpoints for easy client integration

##### Technologies & frameworks to annotate
- Front-end: React, Next.js, TypeScript, Tailwind CSS, i18next
- Back-end: Python, llama-cpp-python, llama.cpp, Flask/FastAPI
- Containerization: Docker, Docker Compose, Dockerfiles (cpu & cuda)
- Orchestration: Kubernetes (kustomize), yamls in deploy/kubernetes
- Storage: Docker volume for /models, browser local storage
- CI/CD: GitHub Actions (.github/workflows/on-push.yml)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User
    Browser["User's Browser"]:::user
    LocalStorage["Browser Local Storage<br>IndexedDB/localStorage"]:::storage

    %% Docker Compose Layer
    subgraph "Docker Compose"
        direction TB
        UI["UI Service<br>Next.js (port 3000)"]:::frontend
        API["API Service<br>FastAPI (port 3001)"]:::backend
        ORCHESTRATION["Orchestration<br>docker-compose.yml"]:::orchestration
        Volume["Model Storage Volume<br>/models"]:::storage
    end

    %% Kubernetes Layer
    subgraph "Kubernetes (namespace: llama)"
        direction TB
        KUBE["Kustomize Manifests"]:::orchestration
    end

    %% Core Inference
    Inference["Inference Engine<br>llama.cpp<br>CPU/GPU"]:::inference

    %% External
    External["Google Search API"]:::external

    %% Flows
    Browser -->|HTTP GET/POST| UI
    UI -->|calls OpenAI API| API
    API -->|calls llama.cpp| Inference
    Inference -->|reads model files| Volume
    API -->|streams completions| UI
    UI -->|save history| LocalStorage
    UI -->|optional HTTP| External

    UI --> ORCHESTRATION
    API --> ORCHESTRATION
    UI --> KUBE
    API --> KUBE

    %% Click Events
    click UI "https://github.com/getumbrel/llama-gpt/tree/master/ui/"
    click API "https://github.com/getumbrel/llama-gpt/blob/master/api/run.sh"
    click Inference "https://github.com/getumbrel/llama-gpt/tree/master/cuda/"
    click Volume "https://github.com/getumbrel/llama-gpt/tree/master/models/"
    click ORCHESTRATION "https://github.com/getumbrel/llama-gpt/blob/master/docker-compose.yml"
    click KUBE "https://github.com/getumbrel/llama-gpt/blob/master/deploy/kubernetes/kustomization.yaml"

    %% Styles
    classDef user fill:#cce5ff,stroke:#3385ff,color:#000
    classDef frontend fill:#ccefff,stroke:#66b3ff,color:#000
    classDef backend fill:#d5f5e3,stroke:#2ecc71,color:#000
    classDef inference fill:#fcf3cf,stroke:#f1c40f,color:#000
    classDef storage fill:#e8e8e8,stroke:#b3b3b3,color:#000
    classDef orchestration fill:#f0f0f0,stroke:#999,color:#000
    classDef external fill:#f2f2f2,stroke:#888,color:#000
```
