How chanyub/whisperx works
Project Type and Purpose
• WhisperX is an open-source Python package and CLI tool for improving timestamp accuracy of Whisper ASR outputs via phoneme-level forced alignment.
• It integrates three key subsystems: Whisper ASR transcription, phoneme-based ASR alignment (e.g. wav2vec2.0), and optional modules (VAD filtering, diarization).
File Structure
Top-level:
• setup.py / requirements.txt – package metadata & dependencies
• whisperx/ – Python source tree
• examples/ – sample media files and demo outputs
• tests/ – unit tests for audio processing, normalization, tokenization, transcription
Core whisperx package:
• __main__.py – CLI entry point
• audio.py – audio I/O, format conversion
• model.py – loading Whisper and alignment models
• transcribe.py – overall pipeline orchestration
• alignment.py – forced alignment logic
• decoding.py – token‐to‐text decoding routines
• tokenizer.py – text tokenization & vocab assets
• normalizers/ – text normalization rules per language
• vad.py – voice activity detection wrapper (pyannote)
• diarize.py – speaker diarization wrapper (pyannote)
• utils.py – shared helper functions
• assets/ – static assets (tokenizer vocab, merges, mel filters)
Extract Insights from the README
• Core flow: CLI → load Whisper model → transcribe → (optional) VAD → load alignment model → align segments → generate word/character time-stamps → optional diarization → subtitle outputs.
• External services/models: OpenAI Whisper, Hugging Face Hub (wav2vec2.0, pyannote).
Main Components & Interactions
a. User / CLI Frontend
• whisperx CLI (entry in __main__.py)
• Python API (via import whisperx)
b. Audio Input & Preprocessing
• audio.py – load file, resample, convert to numpy/tensor
• vad.py (optional) – call pyannote VAD model → split audio into speech segments
c. Whisper ASR Module
• model.py (load_model) → Whisper transformer model
• transcribe.py – generate transcript with utterance-level timestamps
d. Aligner Subsystem
• model.py (load_align_model) → phoneme ASR model (wav2vec2 pipelines)
• tokenizer.py + assets – convert text to phoneme tokens
• alignment.py – forced alignment algorithm aligns tokens to audio
• normalizers/ – cleanse and normalize transcripts before alignment
• decoding.py – post-process alignment outputs
e. Diarization (optional)
• diarize.py – speaker segmentation via pyannote
• merges diarization labels with aligned segments
f. Output Generator
• subtitle (.srt, .ass, .vtt) writer in transcribe.py/util → word_segments, char_segments
• JSON/Python result dict for programmatic use
Architectural Patterns & Design Principles
• Pipeline Architecture: sequential processing stages, each in its own module
• Separation of Concerns: audio I/O, ASR, alignment, normalization, diarization are decoupled
• Dependency Injection: models loaded via factory functions (load_model, load_align_model)
• Extensibility Points:
• New normalizers (add to normalizers/)
• Additional phoneme ASR models (via config or HF pipelines)
• Custom output formats (extend subtitle writer)