How madroidmaq/mlx-omni-server works
Project type and purpose
• MLX Omni Server is a Python-based local inference server implementing OpenAI-compatible REST endpoints (chat, audio, images, embeddings).
• It’s built with FastAPI on Apple Silicon using Apple’s MLX framework, plus Hugging Face for model management.
Top-level architectural layers
• CLIENT LAYER: External callers—curl, OpenAI SDK clients, TestClient (in-process).
• API LAYER: FastAPI HTTP server (main.py + middleware + routers).
• SERVICE LAYER: Individual domain services (chat, embeddings, images, STT, TTS).
• INFERENCE LAYER: MLX framework wrappers, tokenizer & logit processors, third-party models (diffusionkit, whisper, f5-tts-mlx, mlx-embeddings).
• INFRASTRUCTURE LAYER: Local filesystem (model cache), Apple Silicon hardware acceleration, Hugging Face model hub.
Main components & their responsibilities
a. FastAPI Server
• main.py: application entry point
• middleware/logging.py: request/response logging
• routers.py: root router aggregator
b. Domain Routers (src/mlx_omni_server/*/router.py)
• chat.router
• embeddings.router
• images.router
• stt.router
• tts.router
c. Domain Services
• chat/models_service.py, chat/mlx_model.py, chat/outlines_logits_processor.py, prompt_cache.py, stop_tokens_checker.py
• embeddings/embeddings_service.py
• images/images_service.py
• stt/stt.py + whisper_model.py
• tts/tts.py + tts_service.py
d. Inference Tools
• chat/tools (tokenizers, hugging_face download logic, Llama3, Mistral wrappers)
• diffusionkit (external) for images
• whisper for speech-to-text
• f5-tts-mlx for text-to-speech
• mlx-embeddings for embeddings
e. External Services
• Hugging Face model hub (download on demand)
• Local model cache (filesystem)
Component interactions
• Clients → HTTP(S) request → FastAPI Server
• FastAPI routes incoming REST calls to corresponding domain router
• Router validates request against Pydantic schema (schema.py)
• Router invokes domain service
• Domain service calls underlying MLX wrapper or external model API
• Model inference runs on Apple Silicon (MLX + hardware acceleration)
• Inference result flows back through service → router → HTTP response → client
Architectural patterns & principles
• Layered architecture: clear separation between transport (HTTP), routing, business logic (services), and inference internals.
• Modularity: each AI capability lives in its own package.
• Dependency injection (FastAPI): routers register services on startup.
• Open/Closed: domain services can be extended with new model providers.
• Cache & reuse: prompt_cache for repeated chat prompts.
Final checklist
• All endpoints (/v1/chat/completions, /v1/audio, /v1/images, /v1/embeddings, /v1/models) are represented.
• Core modules (chat/mlx tools, whisper, diffusionkit, mlx-embeddings) are shown.
• Client variants (curl, OpenAI SDK, TestClient) are illustrated.