How mahmoudashraf97/whisper-diarization works
Project type and purpose
• This is an open-source, Python-based CLI tool for speaker diarization built on top of OpenAI Whisper and NVIDIA NeMo.
• Purpose: ingest an audio file, separate vocals, transcribe speech, align timestamps, detect speech segments, extract speaker embeddings, and assign speaker labels to each word.
File structure
• Top-level orchestration scripts:
• diarize.py – sequential pipeline
• diarize_parallel.py – experimental parallel execution of ASR & diarization
• helpers.py – shared utility functions
• Configuration folder:
• nemo_msdd_configs/*.yaml – model parameters for general, meeting, and telephonic diarization
• Pipeline modules:
• nemo_process.py – NeMo VAD & embedding orchestration
• diarize.py / diarize_parallel.py – main entry points
• Dependencies & environment:
• requirements.txt & constraints.txt – Python libs (whisper, nemo-toolkit, demucs, ctc-forged-aligner)
• FFmpeg & Cython prerequisites
• Tests:
• tests/assets/test.opus – example audio
• .github/workflows/test_run.yml – CI pipeline
README for additional insights
• Core stages: source separation (Demucs), ASR (Whisper), forced alignment (ctc-forced-aligner), VAD (MarbleNet), speaker embedding (TitaNet), punctuation realignment.
• Two modes: sequential vs. “parallel” execution for high-VRAM systems.
• Future extensibility: CLI argument enhancements, overlapping speaker handling.
Essential elements to capture
• Entry point scripts and their modes (sequential vs parallel).
• Config folder and model YAMLs.
• Data artifacts at each stage (audio, transcript, timestamps, embeddings).
• Final output format (annotated transcript).