openformosa/bluemagpie-tts

1. Repository purpose

BlueMagpie-TTS is an inference-focused text-to-speech package. Its core job is to turn text into 48 kHz mono speech, with three conditioning paths: plain TTS, voice cloning from reference audio, and speaker-vector conditioning. The same model family also supports streaming output and a batched serving engine.

2. Top-level package shape

The main Python package is src/bluemagpie. The public entry point is bluemagpie.__init__, which exposes the user-facing model APIs. The rest of the package is split into model logic, conditioning/loading helpers, and two execution backends: a standard PyTorch path and an Apple Silicon MLX path.

3. Core inference pipeline

The central flow is: input text → tokenizer/semantic language model → conditioning assembly → acoustic generation → waveform output. The acoustic side is the real compute hotspot: the README and file layout indicate a diffusion-style decoder stack (LocEnc, LocDiT, AudioVAE) rather than a single monolithic decoder. The model supports both full generation and streaming/chunked generation.

4. Model composition boundary

The main model logic lives in src/bluemagpie/model.py, with supporting pieces in adapter.py, conditioning.py, centroid.py, tslm.py, loading.py, and config.py. This suggests a layered design:

- loading/config handle checkpoint and runtime setup,

- conditioning/centroid handle speaker and reference-audio inputs,

- adapter/tslm bridge the text side into the speech stack,

- model.py orchestrates the full end-to-end forward path.

5. Text and speaker conditioning paths

The repo explicitly supports four input modes through one generation interface:

- text only,

- text + prompt text/audio for continuation,

- text + reference wav for voice cloning,

- text + speaker centroid for direct voice control.

These all converge into the same generation pipeline, but the conditioning inputs are assembled differently before decoding.

6. Vendorized acoustic implementation

src/bluemagpie/_vendor contains vendored acoustic model code derived from VoxCPM. This is an important boundary: the project does not require a separate install for the acoustic submodules because they are embedded in-repo. That makes the package self-contained on the speech-decoder side while still depending on external text-model assets.

7. Serving subsystem

src/bluemagpie/serving is a separate runtime for throughput-oriented inference. Its main pieces are:

- engine.py: the continuous-batching engine,

- runner.py: execution loop,

- cache.py: sequence/state management,

- streaming.py: incremental output handling,

- barbet_batch.py and minicpm_batch.py: batched text-model execution paths,

- accel.py and _caps.py: device capability and optimization selection.

Architecturally, this layer sits above the same model components but changes scheduling: it batches multiple requests and lets new ones join mid-decode.

8. Serving data flow

In serving mode, each request carries the same conditioning fields as generate, plus an optional seed for determinism. Requests are queued into the engine, batched across active sequences, and then emitted either as complete outputs (run) or incremental chunks (stream). The engine is designed to preserve single-sample numerical behavior while improving throughput by sharing batch work.

9. MLX backend for Apple Silicon

src/bluemagpie/mlx is a full alternate implementation for Apple Silicon. It mirrors the PyTorch stack with MLX-native modules for the semantic model, local conditioning, diffusion decoder, and audio decoder. The design is intentionally parallel to the torch path so parity tests can verify that outputs remain aligned while running on Metal/Apple GPU.

10. Runtime targets and external dependencies

The project is primarily Torch-based, with optional MLX acceleration. Hugging Face is used for model distribution and tokenizer loading; transformers supplies the tokenizer interface. Speaker centroid extraction can optionally use speechbrain via the helper script. The README also notes an external barbet package fetched from GitHub, which is the text-semantic language model dependency.

11. Utility scripts

scripts/extract_speaker_centroid.py is the offline preprocessing tool for voice cloning: it turns reference audio into a 192-dim speaker vector. scripts/bench_rtf.py measures real-time factor, which is relevant because the architecture’s main performance tradeoff is around decoder throughput rather than the text model.

12. Verification strategy

The test suite is architecture-revealing:

- test_model_smoke.py checks end-to-end model loading and generation,

- test_loading.py checks checkpoint/tokenizer loading,

- test_step_equivalence.py checks stepwise numerical consistency,

- test_serving_parity.py checks serving engine behavior against the base model,

- test_mlx_parity.py checks MLX vs PyTorch parity.

This shows the repo’s main quality boundary is not training, but cross-backend inference equivalence.

13. Documentation layout

docs/從_HuggingFace_使用模型.md and the DESIGN.md files under mlx and serving indicate the project has two explicit alternative execution architectures documented separately from the core model. That separation matters: the repo is organized around one model with multiple execution/runtime surfaces, not around a single application stack.