# madroidmaq/mlx-omni-server architecture

> Architecture diagram of the GitHub repository madroidmaq/mlx-omni-server, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/madroidmaq/mlx-omni-server
- Repository: https://github.com/madroidmaq/mlx-omni-server

## Overview

##### Project type and purpose
- MLX Omni Server is a Python-based local inference server implementing OpenAI-compatible REST endpoints (chat, audio, images, embeddings).
- It’s built with FastAPI on Apple Silicon using Apple’s MLX framework, plus Hugging Face for model management.

##### Top-level architectural layers
- CLIENT LAYER: External callers—curl, OpenAI SDK clients, TestClient (in-process).
- API LAYER: FastAPI HTTP server (main.py + middleware + routers).
- SERVICE LAYER: Individual domain services (chat, embeddings, images, STT, TTS).
- INFERENCE LAYER: MLX framework wrappers, tokenizer & logit processors, third-party models (diffusionkit, whisper, f5-tts-mlx, mlx-embeddings).
- INFRASTRUCTURE LAYER: Local filesystem (model cache), Apple Silicon hardware acceleration, Hugging Face model hub.

##### Main components & their responsibilities
a. FastAPI Server
- main.py: application entry point
- middleware/logging.py: request/response logging
- routers.py: root router aggregator
b. Domain Routers (src/mlx_omni_server/*/router.py)
- chat.router
- embeddings.router
- images.router
- stt.router
- tts.router
c. Domain Services
- chat/models_service.py, chat/mlx_model.py, chat/outlines_logits_processor.py, prompt_cache.py, stop_tokens_checker.py
- embeddings/embeddings_service.py
- images/images_service.py
- stt/stt.py + whisper_model.py
- tts/tts.py + tts_service.py
d. Inference Tools
- chat/tools (tokenizers, hugging_face download logic, Llama3, Mistral wrappers)
- diffusionkit (external) for images
- whisper for speech-to-text
- f5-tts-mlx for text-to-speech
- mlx-embeddings for embeddings
e. External Services
- Hugging Face model hub (download on demand)
- Local model cache (filesystem)

##### Component interactions
- Clients → HTTP(S) request → FastAPI Server
- FastAPI routes incoming REST calls to corresponding domain router
- Router validates request against Pydantic schema (schema.py)
- Router invokes domain service
- Domain service calls underlying MLX wrapper or external model API
- Model inference runs on Apple Silicon (MLX + hardware acceleration)
- Inference result flows back through service → router → HTTP response → client

##### Architectural patterns & principles
- Layered architecture: clear separation between transport (HTTP), routing, business logic (services), and inference internals.
- Modularity: each AI capability lives in its own package.
- Dependency injection (FastAPI): routers register services on startup.
- Open/Closed: domain services can be extended with new model providers.
- Cache & reuse: prompt_cache for repeated chat prompts.

##### Final checklist
- All endpoints (/v1/chat/completions, /v1/audio, /v1/images, /v1/embeddings, /v1/models) are represented.
- Core modules (chat/mlx tools, whisper, diffusionkit, mlx-embeddings) are shown.
- Client variants (curl, OpenAI SDK, TestClient) are illustrated.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Layer
    subgraph "Client Layer"
        CLI["curl / OpenAI SDK"]:::client
        TC["TestClient (in-process)"]:::client
    end

    %% API Layer
    subgraph "API Layer"
        MAIN["FastAPI Server"]:::api
        LogM["Logging Middleware"]:::api
        LoggerU["Shared Logger"]:::utility
        RootR["Root Router"]:::router
    end

    %% Domain Layer
    subgraph "Domain Layer"
        subgraph "Routers"
            ChatR["Chat Router"]:::router
            EmbR["Embeddings Router"]:::router
            ImgR["Images Router"]:::router
            STTR["STT Router"]:::router
            TTSR["TTS Router"]:::router
        end
        subgraph "Services"
            ChatS["Chat Service"]:::service
            EmbS["Embeddings Service"]:::service
            ImgS["Images Service"]:::service
            STTS["STT Service"]:::service
            TTSS["TTS Service"]:::service
        end
    end

    %% Inference Layer
    subgraph "Inference Layer"
        subgraph "Chat Inference Tools"
            Token["Tokenizer"]:::infra
            HFDownload["HF Download Logic"]:::infra
            Llama3["Llama3 Wrapper"]:::infra
            Mistral["Mistral Wrapper"]:::infra
            ToolsUtil["Tools Utils"]:::infra
        end
        Diffusion["diffusionkit"]:::infra
        Whisper["whisper"]:::infra
        F5TTS["f5-tts-mlx"]:::infra
        MlxEmb["mlx-embeddings"]:::infra
    end

    %% External Layer
    subgraph "External Layer"
        HFHub["Hugging Face Model Hub"]:::external
        Cache["Local Model Cache"]:::external
    end

    %% Connections
    CLI -->|"HTTP/REST"| MAIN
    MAIN --> LogM
    LogM --> RootR
    TC -->|"In-process"| RootR

    RootR --> ChatR
    RootR --> EmbR
    RootR --> ImgR
    RootR --> STTR
    RootR --> TTSR

    ChatR --> ChatS
    EmbR --> EmbS
    ImgR --> ImgS
    STTR --> STTS
    TTSR --> TTSS

    ChatS --> Token
    ChatS --> HFDownload
    ChatS --> Llama3
    ChatS --> Mistral
    ChatS --> ToolsUtil

    EmbS --> MlxEmb
    ImgS --> Diffusion
    STTS --> Whisper
    TTSS --> F5TTS

    Token --> Cache
    HFDownload --> HFHub
    Llama3 --> Cache
    Mistral --> Cache
    MlxEmb --> HFHub
    Diffusion --> HFHub
    Whisper --> HFHub
    F5TTS --> HFHub

    RootR --> LoggerU

    %% Click Events
    click MAIN "https://github.com/madroidmaq/mlx-omni-server/blob/main/src/mlx_omni_server/main.py"
    click LogM "https://github.com/madroidmaq/mlx-omni-server/blob/main/src/mlx_omni_server/middleware/logging.py"
    click LoggerU "https://github.com/madroidmaq/mlx-omni-server/blob/main/src/mlx_omni_server/utils/logger.py"
    click RootR "https://github.com/madroidmaq/mlx-omni-server/blob/main/src/mlx_omni_server/routers.py"
    click ChatR "https://github.com/madroidmaq/mlx-omni-server/blob/main/src/mlx_omni_server/chat/router.py"
    click ChatS "https://github.com/madroidmaq/mlx-omni-server/blob/main/src/mlx_omni_server/chat/models/models_service.py"
    click Token "https://github.com/madroidmaq/mlx-omni-server/blob/main/src/mlx_omni_server/chat/mlx/tools/chat_tokenizer.py"
    click HFDownload "https://github.com/madroidmaq/mlx-omni-server/blob/main/src/mlx_omni_server/chat/mlx/tools/hugging_face.py"
    click Llama3 "https://github.com/madroidmaq/mlx-omni-server/blob/main/src/mlx_omni_server/chat/mlx/tools/llama3.py"
    click Mistral "https://github.com/madroidmaq/mlx-omni-server/blob/main/src/mlx_omni_server/chat/mlx/tools/mistral.py"
    click ToolsUtil "https://github.com/madroidmaq/mlx-omni-server/blob/main/src/mlx_omni_server/chat/mlx/tools/utils.py"
    click EmbR "https://github.com/madroidmaq/mlx-omni-server/blob/main/src/mlx_omni_server/embeddings/router.py"
    click EmbS "https://github.com/madroidmaq/mlx-omni-server/blob/main/src/mlx_omni_server/embeddings/embeddings_service.py"
    click ImgR "https://github.com/madroidmaq/mlx-omni-server/blob/main/src/mlx_omni_server/images/images.py"
    click ImgS "https://github.com/madroidmaq/mlx-omni-server/blob/main/src/mlx_omni_server/images/images_service.py"
    click STTR "https://github.com/madroidmaq/mlx-omni-server/blob/main/src/mlx_omni_server/stt/stt.py"
    click Whisper "https://github.com/madroidmaq/mlx-omni-server/blob/main/src/mlx_omni_server/stt/whisper_model.py"
    click TTSR "https://github.com/madroidmaq/mlx-omni-server/blob/main/src/mlx_omni_server/tts/tts.py"
    click TTSS "https://github.com/madroidmaq/mlx-omni-server/blob/main/src/mlx_omni_server/tts/tts_service.py"

    %% Styles
    classDef client fill:#c3d9ff,stroke:#000
    classDef api fill:#a3c4f3,stroke:#000
    classDef router fill:#cafccf,stroke:#000
    classDef service fill:#dda0dd,stroke:#000
    classDef infra fill:#ffcc99,stroke:#000
    classDef external fill:#cccccc,stroke:#000
    classDef utility fill:#f0e68c,stroke:#000
```
