How displace2024/displace2024_baseline_updated works
Project Type and Purpose
• This is an open-source, research-oriented speech processing toolkit (baseline for DISPLACE2024) that implements three parallel pipelines:
• Speaker Diarization (SD)
• Language Diarization (LD)
• ASR via Google Cloud Speech-to-Text (STT)
• Each pipeline shares a common front-end (data ingestion, speech activity detection) and diverges into specialized back-ends for embedding extraction, clustering/rescoring, or cloud API calls.
High-Level File Structure
• /language_diarization: Python scripts (VoxLingua-based embedding + AHC clustering)
• /speaker_diarization: Kaldi-style shell recipes & Python wrappers (SAD → features → x-vectors → PLDA → Spectral Clustering → VB-HMM)
• /vad_benchmarking: VAD model wrappers and comparison scripts
• /utils, /tools: shared utilities, installation scripts (Kaldi, dscore)
• /requirements: Python dependencies (core, SAD)
Core Components
A. Data Ingestion & Preprocessing
• Raw audio recordings (close-field/dev & far-field/test)
• Configuration files (e.g., mfcc.conf, fbank_16k.conf)
B. Speech Activity Detection (SAD)
• Pyannote-audio model (Python)
• Generates speech/non-speech boundaries
C. Branch 1: Speaker Diarization Pipeline
1. Feature Extraction (MFCC/fbank via Kaldi scripts)
2. X-vector Extraction (Kaldi nnet3 recipes)
3. PLDA Scoring (Kaldi binaries)
4. Spectral Clustering (Python / clustering.py)
5. VB-HMM Resegmentation (local/diarization/VB_resegmentation*.py + shell)
6. Output RTTM & Evaluation (dscore)
D. Branch 2: Language Diarization Pipeline
1. Utterance‐level feature extraction (feat_extr.py)
2. Embedding extraction (ECAPA-TDNN via SpeechBrain)
3. Agglomerative Hierarchical Clustering (clustering.py)
4. Output language‐labeled segments
E. Branch 3: ASR Pipeline
1. Segment filtering (>=2s)
2. Google Cloud STT API calls (speaker_diarization/run.sh override)
3. Transcript concatenation & WER computation
Relationships & Data Flow
• All pipelines start from “Raw Audio → SAD → Segments.”
• The SD and LD pipelines run in parallel on the same segments.
• ASR runs either on the same segments or on full-session audio.
• Intermediate artifacts (features, x-vectors, embeddings) flow between modules.
• External dependencies: Kaldi toolchain, Pyannote, SpeechBrain, Google Cloud API, dscore.
Architectural Patterns & Design Principles
• Modular pipeline stages (each stage is a discrete script or binary)
• Recipe-style orchestration (shell scripts coordinate Kaldi steps)
• Reusability via shared “utils” directory
• Configuration-driven (conf/*.conf files parameterize feature extraction)
• Extensibility: add new clustering or resegmentation methods under local/diarization