# mahmoudashraf97/whisper-diarization architecture

> Architecture diagram of the GitHub repository mahmoudashraf97/whisper-diarization, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mahmoudashraf97/whisper-diarization
- Repository: https://github.com/mahmoudashraf97/whisper-diarization

## Overview

##### Project type and purpose
- This is an open-source, Python-based CLI tool for speaker diarization built on top of OpenAI Whisper and NVIDIA NeMo.
- Purpose: ingest an audio file, separate vocals, transcribe speech, align timestamps, detect speech segments, extract speaker embeddings, and assign speaker labels to each word.

##### File structure
- Top-level orchestration scripts:
- diarize.py – sequential pipeline
- diarize_parallel.py – experimental parallel execution of ASR & diarization
- helpers.py – shared utility functions
- Configuration folder:
- nemo_msdd_configs/*.yaml – model parameters for general, meeting, and telephonic diarization
- Pipeline modules:
- nemo_process.py – NeMo VAD & embedding orchestration
- diarize.py / diarize_parallel.py – main entry points
- Dependencies & environment:
- requirements.txt & constraints.txt – Python libs (whisper, nemo-toolkit, demucs, ctc-forged-aligner)
- FFmpeg & Cython prerequisites
- Tests:
- tests/assets/test.opus – example audio
- .github/workflows/test_run.yml – CI pipeline

##### README for additional insights
- Core stages: source separation (Demucs), ASR (Whisper), forced alignment (ctc-forced-aligner), VAD (MarbleNet), speaker embedding (TitaNet), punctuation realignment.
- Two modes: sequential vs. “parallel” execution for high-VRAM systems.
- Future extensibility: CLI argument enhancements, overlapping speaker handling.

##### Essential elements to capture
- Entry point scripts and their modes (sequential vs parallel).
- Config folder and model YAMLs.
- Data artifacts at each stage (audio, transcript, timestamps, embeddings).
- Final output format (annotated transcript).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Audio Preparation Stage
    subgraph "Audio Prep"
        direction TB
        FFmpeg["FFmpeg"]:::ext
        Demucs["Demucs (Source Separation)"]:::ext
    end

    %% Transcription Stage
    subgraph "Transcription" 
        direction TB
        Whisper["OpenAI Whisper (ASR)"]:::asr
        CTCAlign["ctc-forced-aligner"]:::align
    end

    %% Diarization Stage
    subgraph "Diarization"
        direction TB
        MarbleNet["MarbleNet (VAD)"]:::nemo
        TitaNet["TitaNet (Speaker Embedding)"]:::nemo
    end

    %% Merge & Label Stage
    MergeLabel["Merge & Labeler (helpers.py)"]:::align

    %% Orchestration & Config
    subgraph "Orchestration & Config"
        direction TB
        CLI_Seq["CLI Orchestrator (sequential)"]:::orchestr
        CLI_Par["CLI Orchestrator (parallel)"]:::orchestr
        Helpers["Shared Utilities (helpers.py)"]:::orchestr
        NemoProcess["NeMo Pipeline (nemo_process.py)"]:::orchestr
        ConfigStore["Config Store (nemo_msdd_configs/)"]:::orchestr
        Requirements["requirements.txt"]:::orchestr
        Constraints["constraints.txt"]:::orchestr
        Notebook["Notebook Demo"]:::orchestr
        TestAudio["Test Audio"]:::ext
        CI["CI Workflow"]:::ext
    end

    %% Output
    Output["Annotated Transcript"]:::ext

    %% Data Flows
    FFmpeg -->|".wav → stems"| Demucs
    Demucs --> Whisper
    Whisper -->|"raw transcript"| CTCAlign
    CTCAlign -->|"aligned word timestamps"| MergeLabel

    Demucs --> MarbleNet
    MarbleNet -->|"segment timestamps"| TitaNet
    TitaNet -->|"speaker embeddings"| MergeLabel

    MergeLabel --> Output

    CLI_Seq --> FFmpeg
    CLI_Par --> FFmpeg
    CLI_Seq --> NemoProcess
    CLI_Par --> NemoProcess
    CLI_Seq --> ConfigStore
    CLI_Par --> ConfigStore
    CLI_Seq --> Requirements
    CLI_Seq --> Constraints
    CLI_Par --> Requirements
    CLI_Par --> Constraints
    CLI_Seq --> Output
    CLI_Par --> Output

    Helpers --> MergeLabel
    NemoProcess --> MarbleNet
    NemoProcess --> TitaNet

    Notebook --> Whisper
    Notebook --> NemoProcess

    CI --> TestAudio
    CI --> CLI_Seq

    %% Click Events
    click CLI_Seq "https://github.com/mahmoudashraf97/whisper-diarization/blob/main/diarize.py"
    click CLI_Par "https://github.com/mahmoudashraf97/whisper-diarization/blob/main/diarize_parallel.py"
    click Helpers "https://github.com/mahmoudashraf97/whisper-diarization/blob/main/helpers.py"
    click NemoProcess "https://github.com/mahmoudashraf97/whisper-diarization/blob/main/nemo_process.py"
    click ConfigStore "https://github.com/mahmoudashraf97/whisper-diarization/tree/main/nemo_msdd_configs/"
    click Requirements "https://github.com/mahmoudashraf97/whisper-diarization/blob/main/requirements.txt"
    click Constraints "https://github.com/mahmoudashraf97/whisper-diarization/blob/main/constraints.txt"
    click Notebook "https://github.com/mahmoudashraf97/whisper-diarization/blob/main/Whisper_Transcription_+_NeMo_Diarization.ipynb"
    click TestAudio "https://github.com/mahmoudashraf97/whisper-diarization/blob/main/tests/assets/test.opus"
    click CI "https://github.com/mahmoudashraf97/whisper-diarization/blob/main/.github/workflows/test_run.yml"

    %% Styles
    classDef asr fill:#AED6F1,stroke:#1B4F72,color:#1B2631
    classDef nemo fill:#ABEBC6,stroke:#196F3D,color:#145A32
    classDef align fill:#FAD7A0,stroke:#CA6F1E,color:#7E5109
    classDef orchestr fill:#D5DBDB,stroke:#566573,color:#1B2631
    classDef ext fill:#F2F3F4,stroke:#ABB2B9,color:#515A5A
```
