How blaizzy/mlx-audio works
Project type and purpose
• This is an open-source Python library (“mlx_audio”) for text-to-speech (TTS), speech-to-speech (STS) and speech-to-text (STT), plus an interactive web interface and REST API. It also ships a Swift/Xcode example app demonstrating on-device TTS via Apple’s MLX framework.
Top-level structure
• Directories
• mlx_audio/ – core Python package
• codec/ – audio codec implementations (Descript, EnCodec, Mimi, SNAC, Vocos)
• stt/ – speech-to-text pipelines (Parakeet, Wav2Vec, Whisper)
• sts/ – speech-to-speech pipeline
• tts/ – text-to-speech pipelines and models (Bark, Dia, Kokoro, Llama, OuTeTTS, Sesame, Spark)
• server.py – FastAPI server entrypoint
• examples/ – demo scripts (e.g., Bible audiobook)
• mlx_audio_swift/ – Swift/Xcode TTS app sample
• README.md, setup.py, requirements.txt, CI configs
Key insights from README
• The library exposes both CLI commands (e.g., mlx_audio.server, mlx_audio.tts.generate) and Python APIs.
• Web interface served by the Python server uses Three.js for 3D audio visualization.
• REST API endpoints for /tts, /audio/{filename}, /play, /stop, /open_output_folder.
• Generated audio files persist under ~/.mlx_audio/outputs.
• Swift example app shows on-device inference layers (Kokoro & Orpheus pipelines) via Apple MLX.
Main components
a. Presentation Layer
• CLI Interface
– Commands: mlx_audio.tts.generate, mlx_audio.server
• Web Front-End
– HTML/JS/CSS + Three.js 3D orb visualization
• Swift Client App
– SwiftUI app bundling MLX TTS engines
b. Service Layer
• FastAPI Server (Uvicorn)
– Hosts REST endpoints and serves static web UI
c. Domain Layer (Inference Pipelines)
• TTS Module
– Pipelines and model loaders (KokoroPipeline, Bark, Spark, etc.)
• STS Module
– Voice-to-voice transformation pipeline
• STT Module
– Parakeet, Wav2Vec, Whisper model pipelines
• Codec Module
– Audio preprocessing, quantization, and compression models
d. Data & Storage
• Local File Store
– Output folder (~/.mlx_audio/outputs)
• External Model Repository
– HuggingFace/MLX model hub
e. Platform/Framework
• Python 3.8+ runtime, MLX core library
• FastAPI, Uvicorn for server
• Three.js for visualization
• Swift & Apple MLX framework for macOS app
Relationships and interactions
• CLI → invokes Python library → calls Domain modules → writes to File Store
• Web UI (browser) → HTTP requests → FastAPI Server → Domain modules → returns audio file names/URLs → UI fetches audio via /audio/{filename}
• Server /play and /stop endpoints control playback subsystem in Python
• Domain modules load external weights from HuggingFace or local caches
• Swift app → on-device MLX TTS engines → fetches weight bundles at runtime or ships bundled weights
Architectural patterns & design principles
• Layered Architecture (Presentation – Service – Domain – Data)
• Modular design: each audio modality (TTS, STT, STS, Codec) isolated in its own package
• CLI and REST API share the same Domain implementation (reuse)
• Configurable model loader abstracts over different model backends
• Quantization support decoupled into Codec module