# gitkaz/mlx_gguf_server architecture

> Architecture diagram of the GitHub repository gitkaz/mlx_gguf_server, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gitkaz/mlx_gguf_server
- Repository: https://github.com/gitkaz/mlx_gguf_server

## Overview

##### Project Type and Purpose
- This is a backend microservice for serving multiple LLM models (MLX‐format and GGUF/llama.cpp) on Apple Silicon Macs.
- It exposes a FastAPI HTTP API (endpoints for model loading, text completions, chat, token counting, cache management, audio transcription, TTS) and uses multiprocessing to isolate each loaded model in its own worker process.

##### High-Level File Structure Analysis
- main.py: entrypoint, configures FastAPI, argument parsing (--enable-whisper, --whisper-model).
- llm_process/: logic for managing and invoking LLM inference in worker processes (loader, request queue, response queue, KV‐cache integration, streaming).
- embedding/: subprocess wrapper for embedding tasks (schemas, run_process).
- whisper_stt/: wrapper around MLX Whisper transcription.
- tts/kokoro_tts/: subprocess wrapper for TTS generation.
- utils/: shared utilities (queue management, cache helpers, schemas.py for Pydantic).
- docs/: usage and KV cache design explanatio­n.

##### Key Architectural Patterns
- API Gateway (FastAPI) → Worker Pool (multiprocessing.Process per model)
- Inter‐process FIFO queues for task dispatch and result retrieval
- Plugin-style subprocess modules (embedding, whisper, TTS) using the same FIFO pattern
- Model abstraction layer (mlx vs llama-cpp) inside each worker process
- KV-cache manager to optimize repeated token usage within a session

##### Main Components and Interactions
A. FastAPI HTTP Server
- Receives HTTP requests from clients
- Parses JSON/multipart, selects model_id (default 0)
- Dispatches tasks into the corresponding worker’s request queue
- Awaits result on the worker’s response queue (supports streaming responses)
B. Process Manager (in llm_process/llm_process.py)
- Maintains a mapping: model_id → ProcessInfo(request_queue, response_queue, Process)
- Routes “load,” “unload,” “completions,” “chat,” “token-count,” “cache_memory,” “cache_limit” tasks
C. Worker Process (one per loaded model)
- Initializes the LLM (via MLX or llama-cpp)
- Listens on its request FIFO queue
- Executes the task: inference, token counting, KV-cache operations
- Places result or streaming chunks into the response queue
D. Embedding, Whisper STT, TTS Subprocesses
- Each has its own run_process.py, queue(s), and schema definitions
- Similar pattern: API endpoint → enqueue into embedding/whisper/tts process → retrieve result
E. Shared Components
- utils/kv_cache_utils.py: KV-cache serialization and updates
- utils/utils.py: queue initialization, logging
- schemas.py: Pydantic models for requests/responses
- logger_config.py: unified logging across processes

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Layers
    subgraph "API Layer" 
        direction TB
        Client["Client"]:::external
        API["FastAPI Server"]:::api
    end

    subgraph "Process Management Layer"
        direction TB
        ProcessMgr["Process Manager"]:::process
    end

    subgraph "Worker Layer"
        direction TB
        subgraph "ModelProcess [ID]"
            direction TB
            ReqQ[(request_queue)]:::queue
            WorkerProc["Worker Process"]:::process
            RespQ[(response_queue)]:::queue
            %% Inside Worker
            subgraph "LLM Worker Internals"
                direction TB
                LLMModel["LLM Abstraction & Loader"]:::process
                KVCache["KV-Cache Manager"]:::util
                MlxStream["Streaming Generation (MLX)"]:::process
                TaskResp["Task/Response Definitions"]:::util
                Logger["Logger Config"]:::util
            end
        end
    end

    subgraph "Subservices"
        direction TB
        subgraph "Embedding Service"
            direction TB
            EmbSchema["Embedding Schemas"]:::service
            EmbRunner["Embedding Runner"]:::service
        end
        subgraph "Whisper STT Service"
            direction TB
            Whisper["Whisper Wrapper"]:::service
        end
        subgraph "TTS Service"
            direction TB
            TTSGen["TTS Generation Logic"]:::service
            TTSRun["TTS Runner"]:::service
        end
    end

    subgraph "Shared Utilities & Docs"
        direction TB
        Utils["Queue & Helpers"]:::util
        KVUtils["KV-Cache Utils"]:::util
        Schemas["Global Pydantic Schemas"]:::util
        Docs1["KV_CACHE.md"]:::doc
        Docs2["KV_CACHE-ja.md"]:::doc
    end

    subgraph "External Dependencies"
        direction TB
        MLX["MLX Framework"]:::external
        LlamaCPP["llama-cpp-python"]:::external
        FFmpeg["ffmpeg"]:::external
    end

    %% Connections
    Client -->|HTTP Request| API
    API -->|dispatch tasks| ProcessMgr
    ProcessMgr -->|enqueue| ReqQ
    ReqQ -->|read task| WorkerProc
    WorkerProc -->|use| LLMModel
    WorkerProc -->|use| KVCache
    WorkerProc -->|use| MlxStream
    WorkerProc -->|use| TaskResp
    WorkerProc -->|log via| Logger
    WorkerProc -->|enqueue result| RespQ
    RespQ -->|deliver response| API
    API -->|HTTP Response| Client

    API -->|embed request| EmbSchema
    API -->|embed dispatch| EmbRunner
    API -->|transcribe| Whisper
    API -->|tts request| TTSGen
    TTSGen -->|spawn| TTSRun

    WorkerProc -->|external lib| MLX
    WorkerProc -->|external lib| LlamaCPP
    Whisper -->|uses| FFmpeg

    API -->|util| Utils
    API -->|schemas| Schemas
    ProcessMgr -->|util| Utils
    WorkerProc -->|util| Utils
    WorkerProc -->|kv util| KVUtils

    %% Click events
    click API "https://github.com/gitkaz/mlx_gguf_server/blob/main/main.py"
    click Schemas "https://github.com/gitkaz/mlx_gguf_server/blob/main/schemas.py"
    click ProcessMgr "https://github.com/gitkaz/mlx_gguf_server/blob/main/llm_process/llm_process.py"
    click LLMModel "https://github.com/gitkaz/mlx_gguf_server/blob/main/llm_process/llm_model.py"
    click KVCache "https://github.com/gitkaz/mlx_gguf_server/blob/main/llm_process/kv_cache_manager.py"
    click MlxStream "https://github.com/gitkaz/mlx_gguf_server/blob/main/llm_process/mlx_generate_stream.py"
    click TaskResp "https://github.com/gitkaz/mlx_gguf_server/blob/main/llm_process/task_response.py"
    click Logger "https://github.com/gitkaz/mlx_gguf_server/blob/main/llm_process/logger_config.py"
    click EmbSchema "https://github.com/gitkaz/mlx_gguf_server/blob/main/embedding/embedding_schemas.py"
    click EmbRunner "https://github.com/gitkaz/mlx_gguf_server/blob/main/embedding/run_process.py"
    click Whisper "https://github.com/gitkaz/mlx_gguf_server/blob/main/whisper_stt/whisper.py"
    click TTSGen "https://github.com/gitkaz/mlx_gguf_server/blob/main/tts/kokoro_tts/generate.py"
    click TTSRun "https://github.com/gitkaz/mlx_gguf_server/blob/main/tts/kokoro_tts/run_process.py"
    click Utils "https://github.com/gitkaz/mlx_gguf_server/blob/main/utils/utils.py"
    click KVUtils "https://github.com/gitkaz/mlx_gguf_server/blob/main/utils/kv_cache_utils.py"
    click Docs1 "https://github.com/gitkaz/mlx_gguf_server/blob/main/docs/KV_CACHE.md"
    click Docs2 "https://github.com/gitkaz/mlx_gguf_server/blob/main/docs/KV_CACHE-ja.md"

    %% Styles
    classDef api fill:#D0E8FF,stroke:#0366d6
    classDef process fill:#E2F7E1,stroke:#28A745
    classDef queue fill:#FFE8A1,stroke:#FFB700,stroke-dasharray: 5 5
    classDef service fill:#F3E5F5,stroke:#9C27B0
    classDef util fill:#FFF4E5,stroke:#FF9800
    classDef external fill:#F0F0F0,stroke:#888888,stroke-dasharray: 2 2
    classDef doc fill:#ECEFF1,stroke:#607D8B,stroke-dasharray: 2 2
```
