How microsoft/vibevoice works
Purpose and Entry Points
VibeVoice is a Python research/inference repository for long-form ASR, streaming ASR, and realtime TTS. The active codebase centers on model implementations, Hugging Face-compatible processing, local demos, and optional vLLM serving. Primary runnable entry points are file/realtime demos under demo/, browser/API serving scripts under vllm_plugin/scripts/, and ASR LoRA training under finetuning-asr/.
Model Core
The modular model layer defines configurations and model variants for standard ASR, streaming ASR, and speech generation/inference:
• vibevoice/modular/modeling_vibevoice_asr.py
• vibevoice/modular/modeling_vibevoice_streaming.py
• vibevoice/modular/modeling_vibevoice_streaming_inference.py
Configuration classes separate static model parameters from execution behavior, including streaming-specific settings:
• vibevoice/modular/configuration_vibevoice.py
• vibevoice/modular/configuration_vibevoice_streaming.py
Token and Diffusion Pipeline
VibeVoice represents speech through continuous/acoustic-semantic tokenization and uses an LLM-backed, next-token diffusion architecture. The tokenizer modules bridge raw/audio representations and model token sequences; the diffusion head generates acoustic detail, with DPM scheduling controlling denoising steps:
• vibevoice/modular/modular_vibevoice_tokenizer.py
• vibevoice/modular/modular_vibevoice_diffusion_head.py
• vibevoice/schedule/dpm_solver.py
This is the principal boundary between model-independent audio preparation and neural inference/generation.
Input Processing and Output Semantics
Processor classes convert audio, text/context, hotwords, and streaming chunks into model inputs. ASR processing supports structured transcription outputs—speaker, timestamp, and text—while streaming processing preserves incremental state/chunk boundaries:
• vibevoice/processor/audio_utils.py
• vibevoice/processor/vibevoice_asr_processor.py
• vibevoice/processor/vibevoice_streaming_processor.py
For realtime generation, streaming output delivery is represented by:
• vibevoice/modular/streamer.py
Inference Paths
Direct local inference is exposed through separate scripts for batch ASR, chunked ASR, realtime TTS, and Gradio/FastAPI demonstrations:
• demo/vibevoice_asr_inference_from_file.py
• demo/vibevoice_asr_streaming_inference_from_file.py
• demo/vibevoice_realtime_demo.py
These scripts are integration boundaries: they load externally hosted model weights, invoke processors/models, and render or write transcription/audio results. Demo voice embeddings/assets for streaming TTS reside in demo/voices/streaming_model/.
vLLM Serving Integration
vllm_plugin/ adapts VibeVoice ASR to vLLM’s model and request lifecycle, including streaming execution and HTTP server endpoints:
• vllm_plugin/model.py
• vllm_plugin/asr_streaming.py
• vllm_plugin/asr_streaming_server.py
Operational launch/API entry points are:
• vllm_plugin/scripts/start_server.py
• vllm_plugin/scripts/start_streaming_server.py
• vllm_plugin/scripts/fastapi_asr_streaming_demo_api.py
This is the scalable-serving boundary; FastAPI provides network access and vLLM supplies accelerated inference/runtime scheduling.
Adaptation and External Dependencies
ASR adaptation is isolated in a LoRA workflow, with a toy audio/JSON dataset contract and inference counterpart:
• finetuning-asr/lora_finetune.py
• finetuning-asr/inference_lora.py
• finetuning-asr/toy_dataset/
The repository depends materially on Python ML/audio runtimes declared in pyproject.toml, Hugging Face model distribution/Transformers compatibility, vLLM for optional serving, and Gradio/FastAPI for demo interfaces. Model checkpoints are external Hugging Face artifacts rather than repository contents.
Main entry points: modeling_vibevoice_asr.py ASR and speech models · modeling_vibevoice_streaming_inference.py Streaming inference model · vibevoice_asr_processor.py ASR processor · configuration_vibevoice.py Model configurations
Components 20
Core inference
Shared model, token, and processing pipeline.
- Model configurations · Static and streaming execution settings.
vibevoice/modular/configuration_vibevoice.py - ASR and speech models · Standard ASR, streaming ASR, and speech-generation variants.
vibevoice/modular/modeling_vibevoice_asr.py - Streaming inference model · Streaming-specific inference behavior.
vibevoice/modular/modeling_vibevoice_streaming_inference.py - Speech tokenizer · Bridges audio representations and model token sequences.
vibevoice/modular/modular_vibevoice_tokenizer.py - Diffusion head · Generates acoustic detail from next-token diffusion.
vibevoice/modular/modular_vibevoice_diffusion_head.py - DPM solver · Controls diffusion denoising steps.
vibevoice/schedule/dpm_solver.py - ASR processor · Builds inputs and structured transcription outputs.
vibevoice/processor/vibevoice_asr_processor.py - Streaming processor · Preserves chunk boundaries and incremental state.
vibevoice/processor/vibevoice_streaming_processor.py - Realtime streamer · Delivers generated streaming output.
vibevoice/modular/streamer.py
Local entry points
File and realtime demo workflows.
- Batch ASR demo · Runs file-based transcription.
demo/vibevoice_asr_inference_from_file.py - Chunked ASR demo · Runs streaming ASR from a file.
demo/vibevoice_asr_streaming_inference_from_file.py - Realtime TTS demo · Runs interactive realtime speech generation.
demo/vibevoice_realtime_demo.py
vLLM serving
Accelerated HTTP and streaming serving integration.
- vLLM model adapter · Adapts VibeVoice ASR to vLLM lifecycle.
vllm_plugin/model.py - vLLM streaming ASR · Executes streaming ASR through vLLM.
vllm_plugin/asr_streaming.py - Streaming API server · Exposes streaming ASR endpoints.
vllm_plugin/asr_streaming_server.py - Server launch scripts · Starts standard and streaming serving modes.
vllm_plugin/scripts/start_streaming_server.py
ASR adaptation
LoRA fine-tuning and adapted inference.
- LoRA fine-tuning · Adapts ASR with audio/JSON data.
finetuning-asr/lora_finetune.py - Toy audio/JSON dataset · Example training-data layout.
- LoRA inference · Runs inference with adapted ASR weights.
finetuning-asr/inference_lora.py
Other components
- Hugging Face checkpoints · Externally hosted model weights.
Connections 23
- Model configurations → ASR and speech models: configures
- Model configurations → Streaming inference model: configures
- Speech tokenizer → ASR and speech models: token inputs
- Diffusion head → ASR and speech models: acoustic generation
- DPM solver → Diffusion head: denoising schedule
- ASR processor → ASR and speech models: ASR inputs
- Streaming processor → Streaming inference model: chunked inputs
- Streaming inference model → Realtime streamer: streamed output
- Batch ASR demo → ASR processor: prepares audio
- Batch ASR demo → ASR and speech models: runs inference
- Chunked ASR demo → Streaming processor: processes chunks
- Chunked ASR demo → Streaming inference model: runs inference
- Realtime TTS demo → Streaming inference model: generates speech
- Realtime TTS demo → Realtime streamer: delivers audio
- Hugging Face checkpoints → ASR and speech models: loads weights
- Hugging Face checkpoints → Streaming inference model: loads weights
- vLLM model adapter → ASR and speech models: adapts ASR model
- vLLM streaming ASR → vLLM model adapter: uses adapter
- Streaming API server → vLLM streaming ASR: serves requests
- Server launch scripts → Streaming API server: starts
- Toy audio/JSON dataset → LoRA fine-tuning: training data
- LoRA fine-tuning → LoRA inference: adapted weights
- LoRA inference → ASR processor: prepares inputs