# blaizzy/mlx-audio architecture

> Architecture diagram of the GitHub repository blaizzy/mlx-audio, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/blaizzy/mlx-audio
- Repository: https://github.com/blaizzy/mlx-audio

## Overview

##### Project type and purpose
- This is an open-source Python library (“mlx_audio”) for text-to-speech (TTS), speech-to-speech (STS) and speech-to-text (STT), plus an interactive web interface and REST API. It also ships a Swift/Xcode example app demonstrating on-device TTS via Apple’s MLX framework.

##### Top-level structure
- Directories
- mlx_audio/ – core Python package
- codec/ – audio codec implementations (Descript, EnCodec, Mimi, SNAC, Vocos)
- stt/ – speech-to-text pipelines (Parakeet, Wav2Vec, Whisper)
- sts/ – speech-to-speech pipeline
- tts/ – text-to-speech pipelines and models (Bark, Dia, Kokoro, Llama, OuTeTTS, Sesame, Spark)
- server.py – FastAPI server entrypoint
- examples/ – demo scripts (e.g., Bible audiobook)
- mlx_audio_swift/ – Swift/Xcode TTS app sample
- README.md, setup.py, requirements.txt, CI configs

##### Key insights from README
- The library exposes both CLI commands (e.g., `mlx_audio.server`, `mlx_audio.tts.generate`) and Python APIs.
- Web interface served by the Python server uses Three.js for 3D audio visualization.
- REST API endpoints for `/tts`, `/audio/{filename}`, `/play`, `/stop`, `/open_output_folder`.
- Generated audio files persist under `~/.mlx_audio/outputs`.
- Swift example app shows on-device inference layers (Kokoro & Orpheus pipelines) via Apple MLX.

##### Main components
a. Presentation Layer
- CLI Interface
– Commands: `mlx_audio.tts.generate`, `mlx_audio.server`
- Web Front-End
– HTML/JS/CSS + Three.js 3D orb visualization
- Swift Client App
– SwiftUI app bundling MLX TTS engines
b. Service Layer
- FastAPI Server (Uvicorn)
– Hosts REST endpoints and serves static web UI
c. Domain Layer (Inference Pipelines)
- TTS Module
– Pipelines and model loaders (KokoroPipeline, Bark, Spark, etc.)
- STS Module
– Voice-to-voice transformation pipeline
- STT Module
– Parakeet, Wav2Vec, Whisper model pipelines
- Codec Module
– Audio preprocessing, quantization, and compression models
d. Data & Storage
- Local File Store
– Output folder (`~/.mlx_audio/outputs`)
- External Model Repository
– HuggingFace/MLX model hub
e. Platform/Framework
- Python 3.8+ runtime, MLX core library
- FastAPI, Uvicorn for server
- Three.js for visualization
- Swift & Apple MLX framework for macOS app

##### Relationships and interactions
- CLI → invokes Python library → calls Domain modules → writes to File Store
- Web UI (browser) → HTTP requests → FastAPI Server → Domain modules → returns audio file names/URLs → UI fetches audio via `/audio/{filename}`
- Server `/play` and `/stop` endpoints control playback subsystem in Python
- Domain modules load external weights from HuggingFace or local caches
- Swift app → on-device MLX TTS engines → fetches weight bundles at runtime or ships bundled weights

##### Architectural patterns & design principles
- Layered Architecture (Presentation – Service – Domain – Data)
- Modular design: each audio modality (TTS, STT, STS, Codec) isolated in its own package
- CLI and REST API share the same Domain implementation (reuse)
- Configurable model loader abstracts over different model backends
- Quantization support decoupled into Codec module

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Presentation Layer
    subgraph "Presentation Layer"
        CLI["CLI Interface"]:::presentation
        WebUI["Web Front-End"]:::presentation
        SwiftApp["Swift Client App"]:::presentation
    end

    %% Service Layer
    subgraph "Service Layer"
        FastAPI["FastAPI Server (Uvicorn)"]:::service
    end

    %% Domain Layer
    subgraph "Domain Layer"
        TTS["TTS Module"]:::domain
        STS["STS Module"]:::domain
        STT["STT Module"]:::domain
        Codec["Codec Module"]:::domain
    end

    %% Data & Storage
    subgraph "Data & Storage"
        Outputs["Audio Output Folder"]:::data
        ModelHub["External Model Repository"]:::data
        FileCache["Local File Cache"]:::data
    end

    %% Connections
    CLI -->|"local API"| TTS
    CLI -->|"local API"| STS
    CLI -->|"local API"| STT
    CLI -->|"local API"| Codec

    WebUI -->|"HTTP POST /tts, /sts, /stt, /codec"| FastAPI
    FastAPI -->|"invoke"| TTS
    FastAPI -->|"invoke"| STS
    FastAPI -->|"invoke"| STT
    FastAPI -->|"invoke"| Codec
    FastAPI -->|"serve UI & audio"| WebUI

    TTS -->|"writes (file I/O)"| Outputs
    STS -->|"writes (file I/O)"| Outputs
    STT -->|"writes (file I/O)"| Outputs
    Codec -->|"writes (file I/O)"| Outputs

    TTS -->|"load weights"| ModelHub
    STS -->|"load weights"| ModelHub
    STT -->|"load weights"| ModelHub
    Codec -->|"load weights"| ModelHub

    Outputs -->|"HTTP GET /audio/{filename}"| WebUI

    SwiftApp -->|"on-device MLX"| TTS
    TTS -->|"writes"| FileCache
    FileCache -->|"serves file"| SwiftApp
    TTS -->|"load weights"| ModelHub

    %% Click Events
    click CLI "https://github.com/blaizzy/mlx-audio/blob/main/mlx_audio/tts/generate.py"
    click WebUI "https://github.com/blaizzy/mlx-audio/blob/main/mlx_audio/tts/audio_player.html"
    click WebUI "https://github.com/blaizzy/mlx-audio/blob/main/mlx_audio/tts/audio_player.py"
    click FastAPI "https://github.com/blaizzy/mlx-audio/blob/main/mlx_audio/server.py"
    click TTS "https://github.com/blaizzy/mlx-audio/tree/main/mlx_audio/tts/models/"
    click STS "https://github.com/blaizzy/mlx-audio/blob/main/mlx_audio/sts/voice_pipeline.py"
    click STS "https://github.com/blaizzy/mlx-audio/tree/main/mlx_audio/sts/tests/"
    click STT "https://github.com/blaizzy/mlx-audio/blob/main/mlx_audio/stt/generate.py"
    click STT "https://github.com/blaizzy/mlx-audio/tree/main/mlx_audio/stt/models/"
    click STT "https://github.com/blaizzy/mlx-audio/tree/main/mlx_audio/stt/tests/"
    click Codec "https://github.com/blaizzy/mlx-audio/tree/main/mlx_audio/codec/"
    click Codec "https://github.com/blaizzy/mlx-audio/tree/main/mlx_audio/codec/tests/"
    click SwiftApp "https://github.com/blaizzy/mlx-audio/tree/main/mlx_audio_swift/"

    %% Styles
    classDef presentation fill:#f9b,stroke:#333,stroke-width:1px;
    classDef service fill:#bbf,stroke:#333,stroke-width:1px;
    classDef domain fill:#bfb,stroke:#333,stroke-width:1px;
    classDef data fill:#ddd,stroke:#333,stroke-width:1px;
```
