# openformosa/bluemagpie-tts architecture

> Architecture diagram of the GitHub repository openformosa/bluemagpie-tts, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openformosa/bluemagpie-tts
- Repository: https://github.com/openformosa/bluemagpie-tts

## Overview

1. **Repository purpose**
   BlueMagpie-TTS is an inference-focused text-to-speech package. Its core job is to turn text into 48 kHz mono speech, with three conditioning paths: plain TTS, voice cloning from reference audio, and speaker-vector conditioning. The same model family also supports streaming output and a batched serving engine.

2. **Top-level package shape**
   The main Python package is `src/bluemagpie`. The public entry point is `bluemagpie.__init__`, which exposes the user-facing model APIs. The rest of the package is split into model logic, conditioning/loading helpers, and two execution backends: a standard PyTorch path and an Apple Silicon MLX path.

3. **Core inference pipeline**
   The central flow is: input text → tokenizer/semantic language model → conditioning assembly → acoustic generation → waveform output. The acoustic side is the real compute hotspot: the README and file layout indicate a diffusion-style decoder stack (`LocEnc`, `LocDiT`, `AudioVAE`) rather than a single monolithic decoder. The model supports both full generation and streaming/chunked generation.

4. **Model composition boundary**
   The main model logic lives in `src/bluemagpie/model.py`, with supporting pieces in `adapter.py`, `conditioning.py`, `centroid.py`, `tslm.py`, `loading.py`, and `config.py`. This suggests a layered design:
   - loading/config handle checkpoint and runtime setup,
   - conditioning/centroid handle speaker and reference-audio inputs,
   - adapter/tslm bridge the text side into the speech stack,
   - model.py orchestrates the full end-to-end forward path.

5. **Text and speaker conditioning paths**
   The repo explicitly supports four input modes through one generation interface:
   - text only,
   - text + prompt text/audio for continuation,
   - text + reference wav for voice cloning,
   - text + speaker centroid for direct voice control.
   These all converge into the same generation pipeline, but the conditioning inputs are assembled differently before decoding.

6. **Vendorized acoustic implementation**
   `src/bluemagpie/_vendor` contains vendored acoustic model code derived from VoxCPM. This is an important boundary: the project does not require a separate install for the acoustic submodules because they are embedded in-repo. That makes the package self-contained on the speech-decoder side while still depending on external text-model assets.

7. **Serving subsystem**
   `src/bluemagpie/serving` is a separate runtime for throughput-oriented inference. Its main pieces are:
   - `engine.py`: the continuous-batching engine,
   - `runner.py`: execution loop,
   - `cache.py`: sequence/state management,
   - `streaming.py`: incremental output handling,
   - `barbet_batch.py` and `minicpm_batch.py`: batched text-model execution paths,
   - `accel.py` and `_caps.py`: device capability and optimization selection.
   Architecturally, this layer sits above the same model components but changes scheduling: it batches multiple requests and lets new ones join mid-decode.

8. **Serving data flow**
   In serving mode, each request carries the same conditioning fields as `generate`, plus an optional seed for determinism. Requests are queued into the engine, batched across active sequences, and then emitted either as complete outputs (`run`) or incremental chunks (`stream`). The engine is designed to preserve single-sample numerical behavior while improving throughput by sharing batch work.

9. **MLX backend for Apple Silicon**
   `src/bluemagpie/mlx` is a full alternate implementation for Apple Silicon. It mirrors the PyTorch stack with MLX-native modules for the semantic model, local conditioning, diffusion decoder, and audio decoder. The design is intentionally parallel to the torch path so parity tests can verify that outputs remain aligned while running on Metal/Apple GPU.

10. **Runtime targets and external dependencies**
   The project is primarily Torch-based, with optional MLX acceleration. Hugging Face is used for model distribution and tokenizer loading; `transformers` supplies the tokenizer interface. Speaker centroid extraction can optionally use `speechbrain` via the helper script. The README also notes an external `barbet` package fetched from GitHub, which is the text-semantic language model dependency.

11. **Utility scripts**
   `scripts/extract_speaker_centroid.py` is the offline preprocessing tool for voice cloning: it turns reference audio into a 192-dim speaker vector. `scripts/bench_rtf.py` measures real-time factor, which is relevant because the architecture’s main performance tradeoff is around decoder throughput rather than the text model.

12. **Verification strategy**
   The test suite is architecture-revealing:
   - `test_model_smoke.py` checks end-to-end model loading and generation,
   - `test_loading.py` checks checkpoint/tokenizer loading,
   - `test_step_equivalence.py` checks stepwise numerical consistency,
   - `test_serving_parity.py` checks serving engine behavior against the base model,
   - `test_mlx_parity.py` checks MLX vs PyTorch parity.
   This shows the repo’s main quality boundary is not training, but cross-backend inference equivalence.

13. **Documentation layout**
   `docs/從_HuggingFace_使用模型.md` and the `DESIGN.md` files under `mlx` and `serving` indicate the project has two explicit alternative execution architectures documented separately from the core model. That separation matters: the repo is organized around one model with multiple execution/runtime surfaces, not around a single application stack.

Main entry points: [`src/bluemagpie/model.py`](https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/model.py) (Model orchestrator), [`src/bluemagpie/serving/engine.py`](https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/serving/engine.py) (Serving engine), [`src/bluemagpie/mlx/model_mlx.py`](https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/mlx/model_mlx.py) (MLX model), [`src/bluemagpie/conditioning.py`](https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/conditioning.py) (Conditioning)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core package"]
  node_src_bluemagpie_init["bluemagpie API<br/>package entrypoint<br/>[__init__.py]"]
  node_src_bluemagpie_model["Model orchestrator<br/>inference model<br/>[model.py]"]
  node_src_bluemagpie_loading["Loading<br/>checkpoint/runtime setup<br/>[loading.py]"]
  node_src_bluemagpie_config["Config<br/>[config.py]"]
  node_src_bluemagpie_conditioning["Conditioning<br/>[conditioning.py]"]
  node_src_bluemagpie_centroid(("Centroid<br/>speaker vector<br/>[centroid.py]"))
  node_src_bluemagpie_adapter["Adapter<br/>text bridge<br/>[adapter.py]"]
  node_src_bluemagpie_tslm["TSLM<br/>semantic LM<br/>[tslm.py]"]
  node_src_bluemagpie_vendor["Vendor acoustic<br/>vendored decoder"]
end

subgraph group_runtime["Execution backends"]
  node_src_bluemagpie_serving_engine["Serving engine<br/>continuous batching<br/>[engine.py]"]
  node_src_bluemagpie_serving_runner["Runner<br/>execution loop<br/>[runner.py]"]
  node_src_bluemagpie_serving_cache["Cache<br/>[cache.py]"]
  node_src_bluemagpie_serving_streaming["Streaming<br/>incremental output<br/>[streaming.py]"]
  node_src_bluemagpie_serving_accel["Accel<br/>device selection<br/>[accel.py]"]
  node_src_bluemagpie_mlx_model["MLX model<br/>apple-silicon backend<br/>[model_mlx.py]"]
  node_src_bluemagpie_mlx_decoder["MLX decoder<br/>acoustic stack<br/>[locenc_mlx.py]"]
end

subgraph group_quality["Verification"]
  node_tests_model_smoke["Model smoke<br/>end-to-end test"]
  node_tests_loading["Loading test<br/>loader test<br/>[test_loading.py]"]
  node_tests_step_equivalence["Step parity<br/>numerical test"]
  node_tests_serving_parity["Serving parity<br/>runtime parity test"]
  node_tests_mlx_parity["MLX parity<br/>backend parity test<br/>[test_mlx_parity.py]"]
end

subgraph group_tools["Tools"]
  node_scripts_extract_centroid["Extract centroid<br/>preprocessing script"]
  node_scripts_bench_rtf["Bench RTF<br/>benchmark script<br/>[bench_rtf.py]"]
end

node_src_bluemagpie_init -->|"exports"| node_src_bluemagpie_model
node_src_bluemagpie_model -->|"uses"| node_src_bluemagpie_loading
node_src_bluemagpie_model -->|"reads"| node_src_bluemagpie_config
node_src_bluemagpie_model -->|"assembles"| node_src_bluemagpie_conditioning
node_src_bluemagpie_model -->|"routes text"| node_src_bluemagpie_adapter
node_src_bluemagpie_model -->|"invokes"| node_src_bluemagpie_tslm
node_src_bluemagpie_model -->|"decodes with"| node_src_bluemagpie_vendor
node_src_bluemagpie_conditioning -->|"consumes"| node_src_bluemagpie_centroid
node_src_bluemagpie_serving_engine -->|"drives"| node_src_bluemagpie_model
node_src_bluemagpie_serving_engine -->|"schedules"| node_src_bluemagpie_serving_runner
node_src_bluemagpie_serving_engine -->|"tracks state"| node_src_bluemagpie_serving_cache
node_src_bluemagpie_serving_engine -->|"streams"| node_src_bluemagpie_serving_streaming
node_src_bluemagpie_serving_engine -->|"selects backend"| node_src_bluemagpie_serving_accel
node_src_bluemagpie_mlx_model -->|"composes"| node_src_bluemagpie_mlx_decoder
node_src_bluemagpie_mlx_model -.->|"parity with"| node_src_bluemagpie_model
node_scripts_extract_centroid -->|"produces"| node_src_bluemagpie_centroid
node_scripts_bench_rtf -.->|"measures"| node_src_bluemagpie_model
node_tests_model_smoke -.->|"validates"| node_src_bluemagpie_model
node_tests_loading -.->|"validates"| node_src_bluemagpie_loading
node_tests_step_equivalence -.->|"validates"| node_src_bluemagpie_model
node_tests_serving_parity -.->|"validates"| node_src_bluemagpie_serving_engine
node_tests_mlx_parity -.->|"validates"| node_src_bluemagpie_mlx_model

click node_src_bluemagpie_init "https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/__init__.py"
click node_src_bluemagpie_model "https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/model.py"
click node_src_bluemagpie_loading "https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/loading.py"
click node_src_bluemagpie_config "https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/config.py"
click node_src_bluemagpie_conditioning "https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/conditioning.py"
click node_src_bluemagpie_centroid "https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/centroid.py"
click node_src_bluemagpie_adapter "https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/adapter.py"
click node_src_bluemagpie_tslm "https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/tslm.py"
click node_src_bluemagpie_vendor "https://github.com/openformosa/bluemagpie-tts/tree/main/src/bluemagpie/_vendor"
click node_src_bluemagpie_serving_engine "https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/serving/engine.py"
click node_src_bluemagpie_serving_runner "https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/serving/runner.py"
click node_src_bluemagpie_serving_cache "https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/serving/cache.py"
click node_src_bluemagpie_serving_streaming "https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/serving/streaming.py"
click node_src_bluemagpie_serving_accel "https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/serving/accel.py"
click node_src_bluemagpie_mlx_model "https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/mlx/model_mlx.py"
click node_src_bluemagpie_mlx_decoder "https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/mlx/locenc_mlx.py"
click node_scripts_extract_centroid "https://github.com/openformosa/bluemagpie-tts/blob/main/scripts/extract_speaker_centroid.py"
click node_scripts_bench_rtf "https://github.com/openformosa/bluemagpie-tts/blob/main/scripts/bench_rtf.py"
click node_tests_model_smoke "https://github.com/openformosa/bluemagpie-tts/blob/main/tests/test_model_smoke.py"
click node_tests_loading "https://github.com/openformosa/bluemagpie-tts/blob/main/tests/test_loading.py"
click node_tests_step_equivalence "https://github.com/openformosa/bluemagpie-tts/blob/main/tests/test_step_equivalence.py"
click node_tests_serving_parity "https://github.com/openformosa/bluemagpie-tts/blob/main/tests/test_serving_parity.py"
click node_tests_mlx_parity "https://github.com/openformosa/bluemagpie-tts/blob/main/tests/test_mlx_parity.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_src_bluemagpie_init,node_src_bluemagpie_model,node_src_bluemagpie_loading,node_src_bluemagpie_config,node_src_bluemagpie_conditioning,node_src_bluemagpie_centroid,node_src_bluemagpie_adapter,node_src_bluemagpie_tslm,node_src_bluemagpie_vendor toneBlue
class node_src_bluemagpie_serving_engine,node_src_bluemagpie_serving_runner,node_src_bluemagpie_serving_cache,node_src_bluemagpie_serving_streaming,node_src_bluemagpie_serving_accel,node_src_bluemagpie_mlx_model,node_src_bluemagpie_mlx_decoder toneAmber
class node_tests_model_smoke,node_tests_loading,node_tests_step_equivalence,node_tests_serving_parity,node_tests_mlx_parity toneMint
class node_scripts_extract_centroid,node_scripts_bench_rtf toneRose
```

## Components

### Core package

Primary inference model, conditioning, and shared runtime helpers.

- [bluemagpie API](https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/__init__.py): package entrypoint · Public package surface exposing user-facing model APIs. · `src/bluemagpie/__init__.py`
- [Model orchestrator](https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/model.py): inference model · Coordinates text-to-speech generation end to end. · `src/bluemagpie/model.py`
- [Loading](https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/loading.py): checkpoint/runtime setup · Loads checkpoints, tokenizer assets, and runtime configuration. · `src/bluemagpie/loading.py`
- [Config](https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/config.py): runtime config · Defines model and runtime configuration structures. · `src/bluemagpie/config.py`
- [Conditioning](https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/conditioning.py): conditioning assembly · Builds speaker, prompt, and reference-audio conditioning inputs. · `src/bluemagpie/conditioning.py`
- [Centroid](https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/centroid.py): speaker vector · Speaker-vector extraction and handling for voice control. · `src/bluemagpie/centroid.py`
- [Adapter](https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/adapter.py): text bridge · Bridges text-side representations into the speech model stack. · `src/bluemagpie/adapter.py`
- [TSLM](https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/tslm.py): semantic LM · Semantic text model interface used by the TTS pipeline. · `src/bluemagpie/tslm.py`
- [Vendor acoustic](https://github.com/openformosa/bluemagpie-tts/tree/main/src/bluemagpie/_vendor): vendored decoder · Embedded acoustic model implementation derived from VoxCPM. · `src/bluemagpie/_vendor`

### Execution backends

Alternative inference runtimes for serving and Apple Silicon.

- [Serving engine](https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/serving/engine.py): continuous batching · Coordinates batched multi-request inference and streaming. · `src/bluemagpie/serving/engine.py`
- [Runner](https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/serving/runner.py): execution loop · Drives request execution and scheduling for serving. · `src/bluemagpie/serving/runner.py`
- [Cache](https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/serving/cache.py): state cache · Manages sequence and generation state across active requests. · `src/bluemagpie/serving/cache.py`
- [Streaming](https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/serving/streaming.py): incremental output · Emits partial generation chunks for low-latency serving. · `src/bluemagpie/serving/streaming.py`
- [Accel](https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/serving/accel.py): device selection · Chooses backend acceleration and optimization paths. · `src/bluemagpie/serving/accel.py`
- [MLX model](https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/mlx/model_mlx.py): apple-silicon backend · MLX-native mirror of the main inference stack. · `src/bluemagpie/mlx/model_mlx.py`
- [MLX decoder](https://github.com/openformosa/bluemagpie-tts/blob/main/src/bluemagpie/mlx/locenc_mlx.py): acoustic stack · MLX-native local conditioning, diffusion, and audio decoding modules. · `src/bluemagpie/mlx/locenc_mlx.py`

### Verification

Tests that check loading, parity, and end-to-end behavior.

- [Model smoke](https://github.com/openformosa/bluemagpie-tts/blob/main/tests/test_model_smoke.py): end-to-end test · Checks loading and basic generation behavior. · `tests/test_model_smoke.py`
- [Loading test](https://github.com/openformosa/bluemagpie-tts/blob/main/tests/test_loading.py): loader test · Verifies checkpoint and tokenizer loading. · `tests/test_loading.py`
- [Step parity](https://github.com/openformosa/bluemagpie-tts/blob/main/tests/test_step_equivalence.py): numerical test · Checks stepwise generation consistency. · `tests/test_step_equivalence.py`
- [Serving parity](https://github.com/openformosa/bluemagpie-tts/blob/main/tests/test_serving_parity.py): runtime parity test · Compares serving behavior against the base model. · `tests/test_serving_parity.py`
- [MLX parity](https://github.com/openformosa/bluemagpie-tts/blob/main/tests/test_mlx_parity.py): backend parity test · Checks MLX outputs against the PyTorch path. · `tests/test_mlx_parity.py`

### Tools

Offline scripts for preprocessing and benchmarking.

- [Extract centroid](https://github.com/openformosa/bluemagpie-tts/blob/main/scripts/extract_speaker_centroid.py): preprocessing script · Offline tool that derives speaker vectors from reference audio. · `scripts/extract_speaker_centroid.py`
- [Bench RTF](https://github.com/openformosa/bluemagpie-tts/blob/main/scripts/bench_rtf.py): benchmark script · Measures real-time factor and throughput behavior. · `scripts/bench_rtf.py`

## Connections

- bluemagpie API → Model orchestrator: exports · Public API surfaces the main model interface.
- Model orchestrator → Loading: uses · Model construction depends on checkpoint and tokenizer loading.
- Model orchestrator → Config: reads · Generation and runtime behavior are driven by config.
- Model orchestrator → Conditioning: assembles · Builds conditioning inputs before decoding.
- Model orchestrator → Adapter: routes text · Text-side features are bridged into the speech stack.
- Model orchestrator → TSLM: invokes · Uses the semantic language model for text-to-semantic generation.
- Model orchestrator → Vendor acoustic: decodes with · Hands off to the vendored acoustic implementation.
- Conditioning → Centroid: consumes · Speaker conditioning can be built from extracted centroids.
- Serving engine → Model orchestrator: drives · Serving reuses the same core model for request execution.
- Serving engine → Runner: schedules · Engine delegates execution to the runner loop.
- Serving engine → Cache: tracks state · Maintains active sequence and generation state.
- Serving engine → Streaming: streams · Supports incremental emission of generated audio/text chunks.
- Serving engine → Accel: selects backend · Chooses device-specific acceleration paths.
- MLX model → MLX decoder: composes · MLX runtime mirrors the core acoustic stack with native modules.
- MLX model → Model orchestrator: parity with · Alternate backend is designed to match the main model behavior.
- Extract centroid → Centroid: produces · Offline preprocessing script generates speaker vectors.
- Bench RTF → Model orchestrator: measures · Benchmark targets the generation path and throughput.
- Model smoke → Model orchestrator: validates · Smoke test covers end-to-end loading and inference.
- Loading test → Loading: validates · Ensures checkpoint and tokenizer loading works.
- Step parity → Model orchestrator: validates · Checks numerical equivalence of stepwise generation.
- Serving parity → Serving engine: validates · Compares serving engine behavior to base inference.
- MLX parity → MLX model: validates · Checks MLX backend parity against the PyTorch path.