# chanyub/whisperx architecture

> Architecture diagram of the GitHub repository chanyub/whisperx, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/chanyub/whisperx
- Repository: https://github.com/chanyub/whisperx

## Overview

##### Project Type and Purpose
- WhisperX is an open-source Python package and CLI tool for improving timestamp accuracy of Whisper ASR outputs via phoneme-level forced alignment.
- It integrates three key subsystems: Whisper ASR transcription, phoneme-based ASR alignment (e.g. wav2vec2.0), and optional modules (VAD filtering, diarization).

##### File Structure
Top-level:
- setup.py / requirements.txt – package metadata & dependencies
- whisperx/ – Python source tree
- examples/ – sample media files and demo outputs
- tests/ – unit tests for audio processing, normalization, tokenization, transcription
Core whisperx package:
- __main__.py – CLI entry point
- audio.py – audio I/O, format conversion
- model.py – loading Whisper and alignment models
- transcribe.py – overall pipeline orchestration
- alignment.py – forced alignment logic
- decoding.py – token‐to‐text decoding routines
- tokenizer.py – text tokenization & vocab assets
- normalizers/ – text normalization rules per language
- vad.py – voice activity detection wrapper (pyannote)
- diarize.py – speaker diarization wrapper (pyannote)
- utils.py – shared helper functions
- assets/ – static assets (tokenizer vocab, merges, mel filters)

##### Extract Insights from the README
- Core flow: CLI → load Whisper model → transcribe → (optional) VAD → load alignment model → align segments → generate word/character time-stamps → optional diarization → subtitle outputs.
- External services/models: OpenAI Whisper, Hugging Face Hub (wav2vec2.0, pyannote).

##### Main Components & Interactions
a. User / CLI Frontend
- whisperx CLI (entry in __main__.py)
- Python API (via import whisperx)
b. Audio Input & Preprocessing
- audio.py – load file, resample, convert to numpy/tensor
- vad.py (optional) – call pyannote VAD model → split audio into speech segments
c. Whisper ASR Module
- model.py (load_model) → Whisper transformer model
- transcribe.py – generate transcript with utterance-level timestamps
d. Aligner Subsystem
- model.py (load_align_model) → phoneme ASR model (wav2vec2 pipelines)
- tokenizer.py + assets – convert text to phoneme tokens
- alignment.py – forced alignment algorithm aligns tokens to audio
- normalizers/ – cleanse and normalize transcripts before alignment
- decoding.py – post-process alignment outputs
e. Diarization (optional)
- diarize.py – speaker segmentation via pyannote
- merges diarization labels with aligned segments
f. Output Generator
- subtitle (.srt, .ass, .vtt) writer in transcribe.py/util → word_segments, char_segments
- JSON/Python result dict for programmatic use

##### Architectural Patterns & Design Principles
- Pipeline Architecture: sequential processing stages, each in its own module
- Separation of Concerns: audio I/O, ASR, alignment, normalization, diarization are decoupled
- Dependency Injection: models loaded via factory functions (load_model, load_align_model)
- Extensibility Points:
- New normalizers (add to normalizers/)
- Additional phoneme ASR models (via config or HF pipelines)
- Custom output formats (extend subtitle writer)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Legend
    subgraph "Legend"
        direction LR
        CM["Core Module"]:::core
        ES["External Service/Model"]:::external
        DA["Data Asset"]:::asset
    end

    %% Entry Point
    CLI["CLI / Python API\nwhisperx/__main__.py"]:::core
    click CLI "https://github.com/chanyub/whisperx/blob/main/whisperx/__main__.py"

    %% Preprocessing
    subgraph "Audio Input & Preprocessing"
        direction TB
        AudioLoader["Audio Loader\nwhisperx/audio.py"]:::core
        click AudioLoader "https://github.com/chanyub/whisperx/blob/main/whisperx/audio.py"
        VAD["VAD Wrapper\nwhisperx/vad.py"]:::core
        click VAD "https://github.com/chanyub/whisperx/blob/main/whisperx/vad.py"
        PyannoteVAD["pyannote VAD"]:::external
    end

    CLI -->|"audio file"| AudioLoader
    AudioLoader -->|"audio→tensor"| VAD
    VAD -->|"(optional)"| PyannoteVAD
    PyannoteVAD -->|"speech segments"| AudioLoader

    %% Whisper ASR
    subgraph "Whisper ASR"
        direction TB
        WhisperModel["Load Whisper Model\nwhisperx/model.py"]:::core
        click WhisperModel "https://github.com/chanyub/whisperx/blob/main/whisperx/model.py"
        OpenAIWhisper["OpenAI Whisper"]:::external
        Transcribe["Pipeline Orchestration\nwhisperx/transcribe.py"]:::core
        click Transcribe "https://github.com/chanyub/whisperx/blob/main/whisperx/transcribe.py"
    end

    AudioLoader -->|"raw audio"| WhisperModel
    WhisperModel -->|"model instance"| Transcribe
    Transcribe -->|"utterance-level segments"| OpenAIWhisper
    OpenAIWhisper -->|"transcript + timestamps"| Transcribe

    %% Aligner Subsystem
    subgraph "Aligner Subsystem"
        direction TB
        AlignModel["Load Alignment Model\nwhisperx/model.py"]:::core
        click AlignModel "https://github.com/chanyub/whisperx/blob/main/whisperx/model.py"
        HFModel["wav2vec2 Model (HF)"]:::external
        Normalizer["Text Normalization\nwhisperx/normalizers/"]:::core
        click Normalizer "https://github.com/chanyub/whisperx/tree/main/whisperx/normalizers/"
        Tokenizer["Tokenizer\nwhisperx/tokenizer.py"]:::core
        click Tokenizer "https://github.com/chanyub/whisperx/blob/main/whisperx/tokenizer.py"
        AssetsVocab["Vocab Assets\nwhisperx/assets/gpt2/\nwhisperx/assets/multilingual/"]:::asset
        click AssetsVocab "https://github.com/chanyub/whisperx/tree/main/whisperx/assets/gpt2/"
        click AssetsVocab "https://github.com/chanyub/whisperx/tree/main/whisperx/assets/multilingual/"
        AlignmentLogic["Forced Alignment\nwhisperx/alignment.py"]:::core
        click AlignmentLogic "https://github.com/chanyub/whisperx/blob/main/whisperx/alignment.py"
        Decoding["Decoding Routines\nwhisperx/decoding.py"]:::core
        click Decoding "https://github.com/chanyub/whisperx/blob/main/whisperx/decoding.py"
    end

    Transcribe -->|"raw segments"| Normalizer
    Normalizer -->|"clean text"| Tokenizer
    Tokenizer -->|"phoneme tokens"| AlignmentLogic
    AssetsVocab --> AlignmentLogic
    AlignModel -->|"alignment model"| AlignmentLogic
    HFModel --> AlignModel
    AlignmentLogic -->|"aligned timestamps"| Decoding
    Decoding -->|"word/char segments"| Transcribe

    %% Diarization (optional)
    subgraph "Diarization (optional)"
        direction TB
        Diarizer["Speaker Diarization\nwhisperx/diarize.py"]:::core
        click Diarizer "https://github.com/chanyub/whisperx/blob/main/whisperx/diarize.py"
        PyannoteDiar["pyannote Diarization"]:::external
    end

    AudioLoader -->|""dash""| Diarizer
    Diarizer --> PyannoteDiar
    PyannoteDiar -->|""speaker labels""| Transcribe

    %% Output Generator
    subgraph "Output Generator"
        direction TB
        Subtitle["Subtitle Writer\n(transcribe/util)"]:::core
        Utils["Shared Utils\nwhisperx/utils.py"]:::core
        click Utils "https://github.com/chanyub/whisperx/blob/main/whisperx/utils.py"
        MelFilters["Mel Filters\nwhisperx/assets/mel_filters.npz"]:::asset
        click MelFilters "https://github.com/chanyub/whisperx/blob/main/whisperx/assets/mel_filters.npz"
    end

    Transcribe -->|"final segments"| Subtitle
    Utils --> Subtitle
    MelFilters --> AudioLoader

    %% Metadata & Tests
    setup["setup.py"]:::asset
    click setup "https://github.com/chanyub/whisperx/blob/main/setup.py"
    req["requirements.txt"]:::asset
    click req "https://github.com/chanyub/whisperx/blob/main/requirements.txt"
    examples["Examples\nexamples/"]:::asset
    click examples "https://github.com/chanyub/whisperx/tree/main/examples/"
    tests["Unit Tests\ntests/"]:::asset
    click tests "https://github.com/chanyub/whisperx/tree/main/tests/"

    CLI --> setup
    CLI --> req
    CLI --> examples
    CLI --> tests

    %% Styles
    classDef core fill:#CEF,stroke:#036,stroke-width:1px;
    classDef external fill:#FCF,stroke:#630,stroke-width:1px;
    classDef asset fill:#CFC,stroke:#063,stroke-width:1px;
```
