How gitkaz/mlx_gguf_server works
Project Type and Purpose
• This is a backend microservice for serving multiple LLM models (MLX‐format and GGUF/llama.cpp) on Apple Silicon Macs.
• It exposes a FastAPI HTTP API (endpoints for model loading, text completions, chat, token counting, cache management, audio transcription, TTS) and uses multiprocessing to isolate each loaded model in its own worker process.
High-Level File Structure Analysis
• main.py: entrypoint, configures FastAPI, argument parsing (--enable-whisper, --whisper-model).
• llm_process/: logic for managing and invoking LLM inference in worker processes (loader, request queue, response queue, KV‐cache integration, streaming).
• embedding/: subprocess wrapper for embedding tasks (schemas, run_process).
• whisper_stt/: wrapper around MLX Whisper transcription.
• tts/kokoro_tts/: subprocess wrapper for TTS generation.
• utils/: shared utilities (queue management, cache helpers, schemas.py for Pydantic).
• docs/: usage and KV cache design explanation.
Key Architectural Patterns
• API Gateway (FastAPI) → Worker Pool (multiprocessing.Process per model)
• Inter‐process FIFO queues for task dispatch and result retrieval
• Plugin-style subprocess modules (embedding, whisper, TTS) using the same FIFO pattern
• Model abstraction layer (mlx vs llama-cpp) inside each worker process
• KV-cache manager to optimize repeated token usage within a session
Main Components and Interactions
A. FastAPI HTTP Server
• Receives HTTP requests from clients
• Parses JSON/multipart, selects model_id (default 0)
• Dispatches tasks into the corresponding worker’s request queue
• Awaits result on the worker’s response queue (supports streaming responses)
B. Process Manager (in llm_process/llm_process.py)
• Maintains a mapping: model_id → ProcessInfo(request_queue, response_queue, Process)
• Routes “load,” “unload,” “completions,” “chat,” “token-count,” “cache_memory,” “cache_limit” tasks
C. Worker Process (one per loaded model)
• Initializes the LLM (via MLX or llama-cpp)
• Listens on its request FIFO queue
• Executes the task: inference, token counting, KV-cache operations
• Places result or streaming chunks into the response queue
D. Embedding, Whisper STT, TTS Subprocesses
• Each has its own run_process.py, queue(s), and schema definitions
• Similar pattern: API endpoint → enqueue into embedding/whisper/tts process → retrieve result
E. Shared Components
• utils/kv_cache_utils.py: KV-cache serialization and updates
• utils/utils.py: queue initialization, logging
• schemas.py: Pydantic models for requests/responses
• logger_config.py: unified logging across processes