# lostruins/koboldcpp architecture

> Architecture diagram of the GitHub repository lostruins/koboldcpp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lostruins/koboldcpp
- Repository: https://github.com/lostruins/koboldcpp

## Overview

##### Main Components Overview
The system can be broken down into these major components:

a) Core Engine Components:
- GGML/GGUF Model Processing Engine
- Multiple Model Architecture Support
- GPU Acceleration Backends (CUDA, Vulkan, Metal, OpenCL)
- Text Generation Pipeline

b) Feature Components:
- KoboldAI API Server
- Stable Diffusion Image Generation
- Speech-to-Text Processing
- Model Format Compatibility Layer

c) User Interface Components:
- Web UI
- CLI Interface
- Configuration Management

##### Detailed Component Relationships

Frontend Layer:
- Web Interface (Port 5001)
- Story Management
- Settings Configuration
- Real-time Generation Interface
- CLI Interface
- Command Parameters
- Configuration Options

API Layer:
- KoboldAI API Endpoint
- OpenAI Compatible API (/v1)
- Stable Diffusion API
- WebSocket Connections

Core Processing Layer:
- Model Management
- Model Loading
- Format Conversion
- Quantization
- Text Generation Engine
- Context Management
- Sampling
- Token Processing

Acceleration Layer:
- GPU Backends
- CUDA
- Vulkan
- Metal
- OpenCL
- CPU Optimization
- Threading
- BLAS Operations

Storage Layer:
- Model Files
- Configuration Storage
- Story Persistence
- Cache Management

##### Key Interactions to Highlight

- Model Loading Pipeline:
- Storage -> Model Loading -> Format Conversion -> Runtime Model

- Text Generation Flow:
- User Input -> API -> Text Generation Engine -> GPU/CPU Processing -> Response

- Multi-modal Integration:
- Input -> Router -> Appropriate Engine (Text/Image/Speech) -> Processing -> Output

##### Important Details

- Port configurations (5001 default)
- Supported model formats (GGUF, legacy GGML)
- GPU layer offloading mechanism
- Context size management
- Multiple model architecture support
- Backend selection logic

##### Integration Points

External integration capabilities:
- KoboldAI API compatibility
- OpenAI API compatibility
- Stable Diffusion integration
- Speech-to-text processing

This system design should help developers understand both the high-level architecture and the detailed interactions between components, while maintaining clarity about the project's scope and capabilities.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Frontend
        WebUI["Web Interface\n(Port 5001)"]
        CLI["CLI Interface"]
        SimpleChat["Simple Chat UI"]
        LegacyUI["Legacy UI"]
    end

    subgraph API
        APIServer["KoboldAI API Server"]
        OpenAIAPI["/v1 OpenAI API"]
        WSConnect["WebSocket\nConnections"]
    end

    subgraph Core
        TextGen["Text Generation\nPipeline"]
        ContextMgr["Context Management"]
        Sampling["Sampling System"]
        ModelLoader["Model Loading"]
        BatchProc["Batch Processing"]
    end

    subgraph Acceleration
        CUDA["CUDA Backend"]
        Vulkan["Vulkan Backend"]
        Metal["Metal Backend"]
        OpenCL["OpenCL Backend"]
    end

    subgraph Storage
        ModelFiles[("Model Files")]
        Config[("Configuration")]
        Stories[("Story Data")]
        Cache[("Cache")]
    end

    subgraph MultiModal
        StableDiff["Stable Diffusion"]
        Speech["Speech Processing"]
    end

    %% Connections
    WebUI --> APIServer
    CLI --> APIServer
    SimpleChat --> APIServer
    LegacyUI --> APIServer

    APIServer --> TextGen
    OpenAIAPI --> TextGen
    WSConnect --> TextGen

    TextGen --> ContextMgr
    TextGen --> Sampling
    TextGen --> ModelLoader
    TextGen --> BatchProc

    ModelLoader --> CUDA
    ModelLoader --> Vulkan
    ModelLoader --> Metal
    ModelLoader --> OpenCL

    ModelLoader --> ModelFiles
    APIServer --> Config
    APIServer --> Stories
    TextGen --> Cache

    APIServer --> StableDiff
    APIServer --> Speech

    %% Click Events
    click WebUI "https://github.com/LostRuins/koboldcpp/tree/concedo/examples/server/webui/"
    click SimpleChat "https://github.com/LostRuins/koboldcpp/tree/concedo/examples/server/public_simplechat/"
    click LegacyUI "https://github.com/LostRuins/koboldcpp/tree/concedo/examples/server/public_legacy/"
    click CLI "https://github.com/LostRuins/koboldcpp/blob/concedo/examples/main/main.cpp"
    click APIServer "https://github.com/LostRuins/koboldcpp/blob/concedo/examples/server/server.cpp"
    click TextGen "https://github.com/LostRuins/koboldcpp/tree/concedo/src/"
    click ContextMgr "https://github.com/LostRuins/koboldcpp/blob/concedo/src/llama-context.cpp"
    click Sampling "https://github.com/LostRuins/koboldcpp/blob/concedo/src/llama-sampling.cpp"
    click ModelLoader "https://github.com/LostRuins/koboldcpp/blob/concedo/src/llama-model-loader.cpp"
    click BatchProc "https://github.com/LostRuins/koboldcpp/blob/concedo/src/llama-batch.cpp"
    click CUDA "https://github.com/LostRuins/koboldcpp/tree/concedo/ggml/src/ggml-cuda/"
    click Vulkan "https://github.com/LostRuins/koboldcpp/tree/concedo/ggml/src/ggml-vulkan/"
    click Metal "https://github.com/LostRuins/koboldcpp/tree/concedo/ggml/src/ggml-metal/"
    click OpenCL "https://github.com/LostRuins/koboldcpp/tree/concedo/ggml/src/ggml-opencl/"
    click StableDiff "https://github.com/LostRuins/koboldcpp/tree/concedo/otherarch/sdcpp/"
    click Speech "https://github.com/LostRuins/koboldcpp/tree/concedo/otherarch/whispercpp/"

    %% Styling
    classDef frontend fill:#a8d1ff
    classDef api fill:#98ff98
    classDef core fill:#ffb366
    classDef accel fill:#e6b3ff
    classDef storage fill:#d9d9d9
    classDef multimodal fill:#ffcc99

    class WebUI,CLI,SimpleChat,LegacyUI frontend
    class APIServer,OpenAIAPI,WSConnect api
    class TextGen,ContextMgr,Sampling,ModelLoader,BatchProc core
    class CUDA,Vulkan,Metal,OpenCL accel
    class ModelFiles,Config,Stories,Cache storage
    class StableDiff,Speech multimodal
```
