# microsoft/vibevoice architecture

> Architecture diagram of the GitHub repository microsoft/vibevoice, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/vibevoice
- Repository: https://github.com/microsoft/vibevoice

## Overview

#### Purpose and Entry Points
VibeVoice is a Python research/inference repository for long-form ASR, streaming ASR, and realtime TTS. The active codebase centers on model implementations, Hugging Face-compatible processing, local demos, and optional vLLM serving. Primary runnable entry points are file/realtime demos under `demo/`, browser/API serving scripts under `vllm_plugin/scripts/`, and ASR LoRA training under `finetuning-asr/`.

#### Model Core
The modular model layer defines configurations and model variants for standard ASR, streaming ASR, and speech generation/inference:
- `vibevoice/modular/modeling_vibevoice_asr.py`
- `vibevoice/modular/modeling_vibevoice_streaming.py`
- `vibevoice/modular/modeling_vibevoice_streaming_inference.py`

Configuration classes separate static model parameters from execution behavior, including streaming-specific settings:
- `vibevoice/modular/configuration_vibevoice.py`
- `vibevoice/modular/configuration_vibevoice_streaming.py`

#### Token and Diffusion Pipeline
VibeVoice represents speech through continuous/acoustic-semantic tokenization and uses an LLM-backed, next-token diffusion architecture. The tokenizer modules bridge raw/audio representations and model token sequences; the diffusion head generates acoustic detail, with DPM scheduling controlling denoising steps:
- `vibevoice/modular/modular_vibevoice_tokenizer.py`
- `vibevoice/modular/modular_vibevoice_diffusion_head.py`
- `vibevoice/schedule/dpm_solver.py`

This is the principal boundary between model-independent audio preparation and neural inference/generation.

#### Input Processing and Output Semantics
Processor classes convert audio, text/context, hotwords, and streaming chunks into model inputs. ASR processing supports structured transcription outputs—speaker, timestamp, and text—while streaming processing preserves incremental state/chunk boundaries:
- `vibevoice/processor/audio_utils.py`
- `vibevoice/processor/vibevoice_asr_processor.py`
- `vibevoice/processor/vibevoice_streaming_processor.py`

For realtime generation, streaming output delivery is represented by:
- `vibevoice/modular/streamer.py`

#### Inference Paths
Direct local inference is exposed through separate scripts for batch ASR, chunked ASR, realtime TTS, and Gradio/FastAPI demonstrations:
- `demo/vibevoice_asr_inference_from_file.py`
- `demo/vibevoice_asr_streaming_inference_from_file.py`
- `demo/vibevoice_realtime_demo.py`

These scripts are integration boundaries: they load externally hosted model weights, invoke processors/models, and render or write transcription/audio results. Demo voice embeddings/assets for streaming TTS reside in `demo/voices/streaming_model/`.

#### vLLM Serving Integration
`vllm_plugin/` adapts VibeVoice ASR to vLLM’s model and request lifecycle, including streaming execution and HTTP server endpoints:
- `vllm_plugin/model.py`
- `vllm_plugin/asr_streaming.py`
- `vllm_plugin/asr_streaming_server.py`

Operational launch/API entry points are:
- `vllm_plugin/scripts/start_server.py`
- `vllm_plugin/scripts/start_streaming_server.py`
- `vllm_plugin/scripts/fastapi_asr_streaming_demo_api.py`

This is the scalable-serving boundary; FastAPI provides network access and vLLM supplies accelerated inference/runtime scheduling.

#### Adaptation and External Dependencies
ASR adaptation is isolated in a LoRA workflow, with a toy audio/JSON dataset contract and inference counterpart:
- `finetuning-asr/lora_finetune.py`
- `finetuning-asr/inference_lora.py`
- `finetuning-asr/toy_dataset/`

The repository depends materially on Python ML/audio runtimes declared in `pyproject.toml`, Hugging Face model distribution/Transformers compatibility, vLLM for optional serving, and Gradio/FastAPI for demo interfaces. Model checkpoints are external Hugging Face artifacts rather than repository contents.

Main entry points: [`vibevoice/modular/modeling_vibevoice_asr.py`](https://github.com/microsoft/vibevoice/blob/main/vibevoice/modular/modeling_vibevoice_asr.py) (ASR and speech models), [`vibevoice/modular/modeling_vibevoice_streaming_inference.py`](https://github.com/microsoft/vibevoice/blob/main/vibevoice/modular/modeling_vibevoice_streaming_inference.py) (Streaming inference model), [`vibevoice/processor/vibevoice_asr_processor.py`](https://github.com/microsoft/vibevoice/blob/main/vibevoice/processor/vibevoice_asr_processor.py) (ASR processor), [`vibevoice/modular/configuration_vibevoice.py`](https://github.com/microsoft/vibevoice/blob/main/vibevoice/modular/configuration_vibevoice.py) (Model configurations)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core inference"]
  node_configs["Model configurations<br/>Python configs"]
  node_models["ASR and speech models<br/>Python models"]
  node_streaming_models["Streaming inference model<br/>Python model"]
  node_tokenizer["Speech tokenizer<br/>Token pipeline"]
  node_diffusion["Diffusion head<br/>Neural head"]
  node_scheduler["DPM solver<br/>Denoising scheduler<br/>[dpm_solver.py]"]
  node_asr_processor["ASR processor<br/>Input processor"]
  node_stream_processor["Streaming processor<br/>Input processor"]
  node_streamer["Realtime streamer<br/>Output delivery<br/>[streamer.py]"]
end

subgraph group_local["Local entry points"]
  node_batch_demo["Batch ASR demo<br/>CLI entry point"]
  node_streaming_demo["Chunked ASR demo<br/>CLI entry point"]
  node_realtime_demo["Realtime TTS demo<br/>Demo entry point"]
end

subgraph group_serving["vLLM serving"]
  node_vllm_model["vLLM model adapter<br/>vLLM integration<br/>[model.py]"]
  node_vllm_streaming["vLLM streaming ASR<br/>Streaming executor<br/>[asr_streaming.py]"]
  node_vllm_server["Streaming API server<br/>HTTP server"]
  node_server_launch["Server launch scripts<br/>Operational entry points"]
end

subgraph group_adaptation["ASR adaptation"]
  node_lora_train["LoRA fine-tuning<br/>Training workflow<br/>[lora_finetune.py]"]
  node_toy_dataset[("Toy audio/JSON dataset<br/>Dataset contract")]
  node_lora_inference["LoRA inference<br/>Adapted inference<br/>[inference_lora.py]"]
end

node_checkpoints[("Hugging Face checkpoints<br/>External model artifacts")]

node_configs -->|"configures"| node_models
node_configs -->|"configures"| node_streaming_models
node_tokenizer -->|"token inputs"| node_models
node_diffusion -->|"acoustic generation"| node_models
node_scheduler -->|"denoising schedule"| node_diffusion
node_asr_processor -->|"ASR inputs"| node_models
node_stream_processor -->|"chunked inputs"| node_streaming_models
node_streaming_models -->|"streamed output"| node_streamer
node_batch_demo -->|"prepares audio"| node_asr_processor
node_batch_demo -->|"runs inference"| node_models
node_streaming_demo -->|"processes chunks"| node_stream_processor
node_streaming_demo -->|"runs inference"| node_streaming_models
node_realtime_demo -->|"generates speech"| node_streaming_models
node_realtime_demo -->|"delivers audio"| node_streamer
node_checkpoints -.->|"loads weights"| node_models
node_checkpoints -.->|"loads weights"| node_streaming_models
node_vllm_model -->|"adapts ASR model"| node_models
node_vllm_streaming -->|"uses adapter"| node_vllm_model
node_vllm_server -->|"serves requests"| node_vllm_streaming
node_server_launch -->|"starts"| node_vllm_server
node_toy_dataset -->|"training data"| node_lora_train
node_lora_train -->|"adapted weights"| node_lora_inference
node_lora_inference -->|"prepares inputs"| node_asr_processor

click node_configs "https://github.com/microsoft/vibevoice/blob/main/vibevoice/modular/configuration_vibevoice.py"
click node_models "https://github.com/microsoft/vibevoice/blob/main/vibevoice/modular/modeling_vibevoice_asr.py"
click node_streaming_models "https://github.com/microsoft/vibevoice/blob/main/vibevoice/modular/modeling_vibevoice_streaming_inference.py"
click node_tokenizer "https://github.com/microsoft/vibevoice/blob/main/vibevoice/modular/modular_vibevoice_tokenizer.py"
click node_diffusion "https://github.com/microsoft/vibevoice/blob/main/vibevoice/modular/modular_vibevoice_diffusion_head.py"
click node_scheduler "https://github.com/microsoft/vibevoice/blob/main/vibevoice/schedule/dpm_solver.py"
click node_asr_processor "https://github.com/microsoft/vibevoice/blob/main/vibevoice/processor/vibevoice_asr_processor.py"
click node_stream_processor "https://github.com/microsoft/vibevoice/blob/main/vibevoice/processor/vibevoice_streaming_processor.py"
click node_streamer "https://github.com/microsoft/vibevoice/blob/main/vibevoice/modular/streamer.py"
click node_batch_demo "https://github.com/microsoft/vibevoice/blob/main/demo/vibevoice_asr_inference_from_file.py"
click node_streaming_demo "https://github.com/microsoft/vibevoice/blob/main/demo/vibevoice_asr_streaming_inference_from_file.py"
click node_realtime_demo "https://github.com/microsoft/vibevoice/blob/main/demo/vibevoice_realtime_demo.py"
click node_vllm_model "https://github.com/microsoft/vibevoice/blob/main/vllm_plugin/model.py"
click node_vllm_streaming "https://github.com/microsoft/vibevoice/blob/main/vllm_plugin/asr_streaming.py"
click node_vllm_server "https://github.com/microsoft/vibevoice/blob/main/vllm_plugin/asr_streaming_server.py"
click node_server_launch "https://github.com/microsoft/vibevoice/blob/main/vllm_plugin/scripts/start_streaming_server.py"
click node_lora_train "https://github.com/microsoft/vibevoice/blob/main/finetuning-asr/lora_finetune.py"
click node_lora_inference "https://github.com/microsoft/vibevoice/blob/main/finetuning-asr/inference_lora.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_configs,node_models,node_streaming_models,node_tokenizer,node_diffusion,node_scheduler,node_asr_processor,node_stream_processor,node_streamer toneBlue
class node_batch_demo,node_streaming_demo,node_realtime_demo toneAmber
class node_vllm_model,node_vllm_streaming,node_vllm_server,node_server_launch toneMint
class node_lora_train,node_toy_dataset,node_lora_inference toneRose
class node_checkpoints toneNeutral
```

## Components

### Core inference

Shared model, token, and processing pipeline.

- [Model configurations](https://github.com/microsoft/vibevoice/blob/main/vibevoice/modular/configuration_vibevoice.py): Python configs · Static and streaming execution settings. · `vibevoice/modular/configuration_vibevoice.py`
- [ASR and speech models](https://github.com/microsoft/vibevoice/blob/main/vibevoice/modular/modeling_vibevoice_asr.py): Python models · Standard ASR, streaming ASR, and speech-generation variants. · `vibevoice/modular/modeling_vibevoice_asr.py`
- [Streaming inference model](https://github.com/microsoft/vibevoice/blob/main/vibevoice/modular/modeling_vibevoice_streaming_inference.py): Python model · Streaming-specific inference behavior. · `vibevoice/modular/modeling_vibevoice_streaming_inference.py`
- [Speech tokenizer](https://github.com/microsoft/vibevoice/blob/main/vibevoice/modular/modular_vibevoice_tokenizer.py): Token pipeline · Bridges audio representations and model token sequences. · `vibevoice/modular/modular_vibevoice_tokenizer.py`
- [Diffusion head](https://github.com/microsoft/vibevoice/blob/main/vibevoice/modular/modular_vibevoice_diffusion_head.py): Neural head · Generates acoustic detail from next-token diffusion. · `vibevoice/modular/modular_vibevoice_diffusion_head.py`
- [DPM solver](https://github.com/microsoft/vibevoice/blob/main/vibevoice/schedule/dpm_solver.py): Denoising scheduler · Controls diffusion denoising steps. · `vibevoice/schedule/dpm_solver.py`
- [ASR processor](https://github.com/microsoft/vibevoice/blob/main/vibevoice/processor/vibevoice_asr_processor.py): Input processor · Builds inputs and structured transcription outputs. · `vibevoice/processor/vibevoice_asr_processor.py`
- [Streaming processor](https://github.com/microsoft/vibevoice/blob/main/vibevoice/processor/vibevoice_streaming_processor.py): Input processor · Preserves chunk boundaries and incremental state. · `vibevoice/processor/vibevoice_streaming_processor.py`
- [Realtime streamer](https://github.com/microsoft/vibevoice/blob/main/vibevoice/modular/streamer.py): Output delivery · Delivers generated streaming output. · `vibevoice/modular/streamer.py`

### Local entry points

File and realtime demo workflows.

- [Batch ASR demo](https://github.com/microsoft/vibevoice/blob/main/demo/vibevoice_asr_inference_from_file.py): CLI entry point · Runs file-based transcription. · `demo/vibevoice_asr_inference_from_file.py`
- [Chunked ASR demo](https://github.com/microsoft/vibevoice/blob/main/demo/vibevoice_asr_streaming_inference_from_file.py): CLI entry point · Runs streaming ASR from a file. · `demo/vibevoice_asr_streaming_inference_from_file.py`
- [Realtime TTS demo](https://github.com/microsoft/vibevoice/blob/main/demo/vibevoice_realtime_demo.py): Demo entry point · Runs interactive realtime speech generation. · `demo/vibevoice_realtime_demo.py`

### vLLM serving

Accelerated HTTP and streaming serving integration.

- [vLLM model adapter](https://github.com/microsoft/vibevoice/blob/main/vllm_plugin/model.py): vLLM integration · Adapts VibeVoice ASR to vLLM lifecycle. · `vllm_plugin/model.py`
- [vLLM streaming ASR](https://github.com/microsoft/vibevoice/blob/main/vllm_plugin/asr_streaming.py): Streaming executor · Executes streaming ASR through vLLM. · `vllm_plugin/asr_streaming.py`
- [Streaming API server](https://github.com/microsoft/vibevoice/blob/main/vllm_plugin/asr_streaming_server.py): HTTP server · Exposes streaming ASR endpoints. · `vllm_plugin/asr_streaming_server.py`
- [Server launch scripts](https://github.com/microsoft/vibevoice/blob/main/vllm_plugin/scripts/start_streaming_server.py): Operational entry points · Starts standard and streaming serving modes. · `vllm_plugin/scripts/start_streaming_server.py`

### ASR adaptation

LoRA fine-tuning and adapted inference.

- [LoRA fine-tuning](https://github.com/microsoft/vibevoice/blob/main/finetuning-asr/lora_finetune.py): Training workflow · Adapts ASR with audio/JSON data. · `finetuning-asr/lora_finetune.py`
- **Toy audio/JSON dataset**: Dataset contract · Example training-data layout.
- [LoRA inference](https://github.com/microsoft/vibevoice/blob/main/finetuning-asr/inference_lora.py): Adapted inference · Runs inference with adapted ASR weights. · `finetuning-asr/inference_lora.py`

### Other components

- **Hugging Face checkpoints**: External model artifacts · Externally hosted model weights.

## Connections

- Model configurations → ASR and speech models: configures
- Model configurations → Streaming inference model: configures
- Speech tokenizer → ASR and speech models: token inputs
- Diffusion head → ASR and speech models: acoustic generation
- DPM solver → Diffusion head: denoising schedule
- ASR processor → ASR and speech models: ASR inputs
- Streaming processor → Streaming inference model: chunked inputs
- Streaming inference model → Realtime streamer: streamed output
- Batch ASR demo → ASR processor: prepares audio
- Batch ASR demo → ASR and speech models: runs inference
- Chunked ASR demo → Streaming processor: processes chunks
- Chunked ASR demo → Streaming inference model: runs inference
- Realtime TTS demo → Streaming inference model: generates speech
- Realtime TTS demo → Realtime streamer: delivers audio
- Hugging Face checkpoints → ASR and speech models: loads weights
- Hugging Face checkpoints → Streaming inference model: loads weights
- vLLM model adapter → ASR and speech models: adapts ASR model
- vLLM streaming ASR → vLLM model adapter: uses adapter
- Streaming API server → vLLM streaming ASR: serves requests
- Server launch scripts → Streaming API server: starts
- Toy audio/JSON dataset → LoRA fine-tuning: training data
- LoRA fine-tuning → LoRA inference: adapted weights
- LoRA inference → ASR processor: prepares inputs
