# gpustack/llama-box architecture

> Architecture diagram of the GitHub repository gpustack/llama-box, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gpustack/llama-box
- Repository: https://github.com/gpustack/llama-box

## Overview

##### Main Components Identification

a) Core Server Components:
- HTTP Server Layer (handling REST API endpoints)
- RPC Server Layer (for remote inference)
- Rate Limiter
- Request Handler

b) Model Processing Components:
- Text Model Processor (LLaMA)
- Image Model Processor (Stable Diffusion)
- Vision Model Processor (for multimodal capabilities)
- Embedding Processor
- Reranker Processor

c) Resource Management:
- KV Cache Manager
- GPU Memory Manager
- CPU Thread Manager
- Slot Manager

**[Client Layer]**

- ▼
**[API Gateway Layer]**

- ▼
**[Server Core]**

- [Request Handler]

- [Rate Limiter]
- [Slot Manager]

- [Model Processing Layer]

- [Text Processing]
- [Image Processing]
- [Vision Processing]
- [Embedding Processing]
- [Reranking]

- [Resource Management]

- [Memory Manager]
- [KV Cache]
- [Thread Pool]

##### Additional Elements

Important features:
- Prometheus metrics endpoint
- Health check endpoint
- Vision API integration
- Function calling capability
- Speculative decoding
- Rate limiting

##### Notes

Annotations for:
- Supported platforms/devices
- Memory management strategies
- Threading model
- Cache strategies

Remember to include version information and any specific hardware requirements as notes, as these are crucial for deployment considerations.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Client["Client Layer"]
        HTTP[HTTP Client]
        RPC[RPC Client]
    end

    subgraph Gateway["API Gateway Layer"]
        REST["REST API Endpoints"]
        RPCServer["RPC Server"]:::rpc
        Metrics["Prometheus Metrics"]
        Health["Health Check"]
    end

    subgraph Core["Server Core"]
        Handler["Request Handler"]:::handler
        RateLim["Rate Limiter"]:::ratelim
        SlotMgr["Slot Manager"]
        ParamMgr["Parameter Manager"]:::param
    end

    subgraph Models["Model Processing Layer"]
        Text["Text Processing"]:::text
        Image["Image Generation"]:::image
        Vision["Vision/CLIP"]:::vision
        Embed["Embedding"]:::embed
        Tools["Tool Calling"]:::tools
    end

    subgraph Resources["Resource Management"]
        Memory["Memory Manager"]
        Cache["KV Cache"]:::cache
        ThreadPool["Thread Manager"]:::thread
        GPU["GPU CUDA Manager"]:::gpu
        Metal["Metal Processing"]:::metal
    end

    %% Connections
    Client --> Gateway
    Gateway --> Core
    Core --> Models
    Core --> Resources

    Handler --> RateLim
    Handler --> SlotMgr
    Handler --> ParamMgr

    %% Click events
    click RPCServer "https://github.com/gpustack/llama-box/blob/main/llama-box/rpcserver.hpp"
    click RateLim "https://github.com/gpustack/llama-box/blob/main/llama-box/ratelimiter.hpp"
    click Handler "https://github.com/gpustack/llama-box/blob/main/llama-box/server.cpp"
    click Image "https://github.com/gpustack/llama-box/blob/main/llama-box/stablediffusion.hpp"
    click Text "https://github.com/gpustack/llama-box/tree/main/llama.cpp/"
    click Image "https://github.com/gpustack/llama-box/tree/main/stable-diffusion.cpp/"
    click Embed "https://github.com/gpustack/llama-box/blob/main/llama-box/patches/llama.cpp/embedding.patch"
    click Tools "https://github.com/gpustack/llama-box/blob/main/llama-box/patches/llama.cpp/tool_calling.patch"
    click Vision "https://github.com/gpustack/llama-box/blob/main/llama-box/patches/llama.cpp/clip.patch"
    click GPU "https://github.com/gpustack/llama-box/blob/main/llama-box/patches/llama.cpp/ggml-cuda.patch"
    click ThreadPool "https://github.com/gpustack/llama-box/blob/main/llama-box/patches/llama.cpp/pure_cpu.patch"
    click Metal "https://github.com/gpustack/llama-box/blob/main/llama-box/patches/llama.cpp/ggml-metal.patch"
    click Cache "https://github.com/gpustack/llama-box/blob/main/llama-box/patches/llama.cpp/ngram_cache.patch"
    click ParamMgr "https://github.com/gpustack/llama-box/blob/main/llama-box/param.hpp"

    %% Styling
    classDef default fill:#f9f9f9,stroke:#333,stroke-width:2px
    classDef rpc fill:#a8e6cf,stroke:#333
    classDef handler fill:#dcedc1,stroke:#333
    classDef ratelim fill:#ffd3b6,stroke:#333
    classDef text fill:#ffaaa5,stroke:#333
    classDef image fill:#ff8b94,stroke:#333
    classDef vision fill:#98ddca,stroke:#333
    classDef embed fill:#d4a5a5,stroke:#333
    classDef tools fill:#9a8194,stroke:#333
    classDef cache fill:#ffb6b9,stroke:#333
    classDef thread fill:#fae3d9,stroke:#333
    classDef gpu fill:#bbded6,stroke:#333
    classDef metal fill:#8ac6d1,stroke:#333
    classDef param fill:#b8a9c9,stroke:#333
```
