How getumbrel/llama-gpt works
Project type and purpose
• Full-stack application comprised of:
– A React/Next.js front-end (“ui” folder)
– A Python-based, OpenAI-compatible API back-end (“api” folder, llama-cpp-python)
– Optional GPU acceleration (CUDA Dockerfiles)
– Model storage (local volume)
– Helm/Kubernetes and Docker Compose deployment manifests
File structure
• Top-level directories:
– ui/ : Next.js app, pages, components, hooks, styles, Dockerfile, k8s manifest
– api/ : Shell script to launch the Python API server wrapping llama-cpp
– cuda/ : Dockerfiles and scripts for NVIDIA GPU support
– deploy/kubernetes/ : Kustomize manifests for UI & API Deployments and Services
– models/ : Volume mount point for downloaded model files
• Root scripts: run.sh, run-mac.sh orchestrate Docker Compose setups
• Docker Compose files: cpu (GGML), cuda, mac, etc.
Key insights from README
• The system exposes two HTTP services:
– UI on port 3000 (Next.js SSR + React SPA)
– API on port 3001 (OpenAI-compatible endpoints)
• Models auto-download into /models on first run
• Supports CPU (GGML) or GPU (CUDA) inference
• Kubernetes manifests allow cloud or on-prem deployment
Main components
a. User’s Browser
– React UI (Next.js server)
– Local state & storage (IndexedDB/localStorage for conversation history)
b. UI Service Container
– Next.js server (pages/api proxies, SSR)
– Static assets, i18n, Tailwind CSS
c. API Service Container
– Python Flask/FastAPI (llama-cpp-python) exposing /v1/chat/completions etc.
– Drops in as OpenAI replacement
d. Model Storage Volume
– Shared Docker/K8s volume at /models
– Contains GGML or GGUF quantized weights
e. Inference Engine
– llama.cpp library (CPU) or CUDA-enabled backend
f. External Services (optional)
– Google Search plugin API (pages/api/google.ts)
– (Future) external plugins
g. Orchestration & Deployment
– Docker Compose network connecting ui & api containers
– Kubernetes Deployments + Services under namespace “llama”
Component interactions & data flow
• Browser → UI Service (HTTP GET/POST)
• UI Service → API Service (HTTP REST/OpenAI API calls)
• API Service → llama.cpp (in-process library call)
• llama.cpp → Model files on volume → returns tokens
• API Service → UI Service → Browser (streamed Completions)
• UI Service → Local browser storage (save conversations)
• Optional: UI Service → Google Search Service → UI Service
Architectural patterns & design principles
• Separation of concerns: UI vs. inference API
• Microservice-style deployment (2 containers, shared volume)
• Swap-in inference backends (CPU/GPU builds) via Dockerfiles
• Declarative infra: Docker Compose & Kubernetes Kustomize
• API compatibility: mimic OpenAI endpoints for easy client integration
Technologies & frameworks to annotate
• Front-end: React, Next.js, TypeScript, Tailwind CSS, i18next
• Back-end: Python, llama-cpp-python, llama.cpp, Flask/FastAPI
• Containerization: Docker, Docker Compose, Dockerfiles (cpu & cuda)
• Orchestration: Kubernetes (kustomize), yamls in deploy/kubernetes
• Storage: Docker volume for /models, browser local storage
• CI/CD: GitHub Actions (.github/workflows/on-push.yml)