# displace2024/displace2024_baseline_updated architecture

> Architecture diagram of the GitHub repository displace2024/displace2024_baseline_updated, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/displace2024/displace2024_baseline_updated
- Repository: https://github.com/displace2024/displace2024_baseline_updated

## Overview

##### Project Type and Purpose
- This is an open-source, research-oriented speech processing toolkit (baseline for DISPLACE2024) that implements three parallel pipelines:
- Speaker Diarization (SD)
- Language Diarization (LD)
- ASR via Google Cloud Speech-to-Text (STT)
- Each pipeline shares a common front-end (data ingestion, speech activity detection) and diverges into specialized back-ends for embedding extraction, clustering/rescoring, or cloud API calls.

##### High-Level File Structure
- /language_diarization: Python scripts (VoxLingua-based embedding + AHC clustering)
- /speaker_diarization: Kaldi-style shell recipes & Python wrappers (SAD → features → x-vectors → PLDA → Spectral Clustering → VB-HMM)
- /vad_benchmarking: VAD model wrappers and comparison scripts
- /utils, /tools: shared utilities, installation scripts (Kaldi, dscore)
- /requirements: Python dependencies (core, SAD)

##### Core Components
A. Data Ingestion & Preprocessing
- Raw audio recordings (close-field/dev & far-field/test)
- Configuration files (e.g., mfcc.conf, fbank_16k.conf)
B. Speech Activity Detection (SAD)
- Pyannote-audio model (Python)
- Generates speech/non-speech boundaries
C. Branch 1: Speaker Diarization Pipeline
1. Feature Extraction (MFCC/fbank via Kaldi scripts)
2. X-vector Extraction (Kaldi nnet3 recipes)
3. PLDA Scoring (Kaldi binaries)
4. Spectral Clustering (Python / clustering.py)
5. VB-HMM Resegmentation (local/diarization/VB_resegmentation*.py + shell)
6. Output RTTM & Evaluation (dscore)
D. Branch 2: Language Diarization Pipeline
1. Utterance‐level feature extraction (feat_extr.py)
2. Embedding extraction (ECAPA-TDNN via SpeechBrain)
3. Agglomerative Hierarchical Clustering (clustering.py)
4. Output language‐labeled segments
E. Branch 3: ASR Pipeline
1. Segment filtering (>=2s)
2. Google Cloud STT API calls (speaker_diarization/run.sh override)
3. Transcript concatenation & WER computation

##### Relationships & Data Flow
- All pipelines start from “Raw Audio → SAD → Segments.”
- The SD and LD pipelines run in parallel on the same segments.
- ASR runs either on the same segments or on full-session audio.
- Intermediate artifacts (features, x-vectors, embeddings) flow between modules.
- External dependencies: Kaldi toolchain, Pyannote, SpeechBrain, Google Cloud API, dscore.

##### Architectural Patterns & Design Principles
- Modular pipeline stages (each stage is a discrete script or binary)
- Recipe-style orchestration (shell scripts coordinate Kaldi steps)
- Reusability via shared “utils” directory
- Configuration-driven (conf/*.conf files parameterize feature extraction)
- Extensibility: add new clustering or resegmentation methods under local/diarization

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Pre-processing Layer
    subgraph "Pre-processing"
        direction TB
        RawAudio["Raw Audio Recordings"]:::external
        Configs["Config Files\n(mfcc.conf, fbank_16k.conf)"]:::docs
        DataIngest["Data Ingestion & Preprocessing"]:::python
        SAD["SAD (Pyannote)"]:::python
        Segments["Speech Segments"]:::python
    end

    RawAudio -->|audio| DataIngest
    Configs -.->|configs| DataIngest
    DataIngest --> SAD
    click Configs "https://github.com/displace2024/displace2024_baseline_updated/blob/main/speaker_diarization/conf/mfcc.conf"
    click Configs "https://github.com/displace2024/displace2024_baseline_updated/blob/main/speaker_diarization/conf/fbank_16k.conf"
    SAD --> Segments
    click SAD "https://github.com/displace2024/displace2024_baseline_updated/blob/main/language_diarization/code/pyannote_vad.py"
    click SAD "https://github.com/displace2024/displace2024_baseline_updated/blob/main/vad_benchmarking/VAD.py"
    click SAD "https://github.com/displace2024/displace2024_baseline_updated/blob/main/vad_benchmarking/run_pyannote_SAD.sh"
    click SAD "https://github.com/displace2024/displace2024_baseline_updated/blob/main/speaker_diarization/local/segmentation/detect_speech_activity.sh"

    %% Speaker Diarization Pipeline
    subgraph "Speaker Diarization Pipeline"
        direction TB
        FEat["Feature Extraction\n(MFCC/fbank)"]:::kaldi
        XVec["X-vector Extraction"]:::kaldi
        PLDA["PLDA Scoring"]:::kaldi
        SC["Spectral Clustering"]:::python
        VB["VB-HMM Resegmentation"]:::python
        RTTM["RTTM Output & Evaluation"]:::python
    end

    Segments --> FEat
    click FEat "https://github.com/displace2024/displace2024_baseline_updated/blob/main/speaker_diarization/local/nnet3/xvector/prepare_feats.sh"
    click FEat "https://github.com/displace2024/displace2024_baseline_updated/tree/main/speaker_diarization/steps"
    FEat --> XVec
    click XVec "https://github.com/displace2024/displace2024_baseline_updated/blob/main/speaker_diarization/local/nnet3/xvector/extract_xvectors.sh"
    click XVec "https://github.com/displace2024/displace2024_baseline_updated/blob/main/speaker_diarization/local/nnet3/xvector/extract_xvectors_pytorch.sh"
    click XVec "https://github.com/displace2024/displace2024_baseline_updated/blob/main/speaker_diarization/local/nnet3/xvector/extract_xvectors_withgpu.sh"
    XVec --> PLDA
    click PLDA "https://github.com/displace2024/displace2024_baseline_updated/blob/main/speaker_diarization/local/diarization/nnet3/xvector/score_plda.sh"
    click PLDA "https://github.com/displace2024/displace2024_baseline_updated/blob/main/speaker_diarization/local/diarization/nnet3/xvector/score_plda_energy.sh"
    PLDA --> SC
    click SC "https://github.com/displace2024/displace2024_baseline_updated/blob/main/speaker_diarization/local/diarization/spectral_scoring/run_spectralclustering.py"
    click SC "https://github.com/displace2024/displace2024_baseline_updated/blob/main/speaker_diarization/local/diarization/spectral_scoring/SpectralCluster/spectralcluster/spectral_clusterer.py"
    SC --> VB
    click VB "https://github.com/displace2024/displace2024_baseline_updated/blob/main/speaker_diarization/local/diarization/VB_resegmentation.py"
    click VB "https://github.com/displace2024/displace2024_baseline_updated/blob/main/speaker_diarization/local/diarization/VB_resegmentation.sh"
    click VB "https://github.com/displace2024/displace2024_baseline_updated/blob/main/speaker_diarization/local/diarization/VB_resegmentation_overlap.py"
    click VB "https://github.com/displace2024/displace2024_baseline_updated/blob/main/speaker_diarization/local/diarization/VB_resegmentation_overlap.sh"
    VB --> RTTM
    click RTTM "https://github.com/displace2024/displace2024_baseline_updated/blob/main/speaker_diarization/local/diarization/make_rttm.py"
    click RTTM "https://github.com/displace2024/displace2024_baseline_updated/blob/main/utils/rttm_speaker_id.py"
    click RTTM "https://github.com/displace2024/displace2024_baseline_updated/blob/main/tools/install_dscore.sh"

    %% Language Diarization Pipeline
    subgraph "Language Diarization Pipeline"
        direction TB
        LD_Feat["Utterance-level Feature Extraction"]:::python
        Emb["ECAPA-TDNN Embeddings"]:::external
        AHC["AHC Clustering"]:::python
        LD_Run["Run & Score"]:::python
    end

    Segments --> LD_Feat
    click LD_Feat "https://github.com/displace2024/displace2024_baseline_updated/blob/main/language_diarization/code/feat_extr.py"
    LD_Feat --> Emb
    click Emb "https://github.com/displace2024/displace2024_baseline_updated/blob/main/language_diarization/code/VBx.py"
    Emb --> AHC
    click AHC "https://github.com/displace2024/displace2024_baseline_updated/blob/main/language_diarization/code/clustering.py"
    AHC --> LD_Run
    click LD_Run "https://github.com/displace2024/displace2024_baseline_updated/blob/main/language_diarization/code/run_all.sh"
    click LD_Run "https://github.com/displace2024/displace2024_baseline_updated/blob/main/language_diarization/code/score.py"
    click LD_Run "https://github.com/displace2024/displace2024_baseline_updated/blob/main/language_diarization/code/seg2kaldi.py"

    %% ASR Pipeline
    subgraph "ASR Pipeline"
        direction TB
        Filter["Segment Filtering (>=2s)"]:::python
        GCloud["Google Cloud STT"]:::external
        WER["Transcript & WER"]:::python
    end

    Segments --> Filter
    Filter --> GCloud
    click GCloud "https://github.com/displace2024/displace2024_baseline_updated/blob/main/speaker_diarization/run.sh"
    GCloud --> WER
    click WER "https://github.com/displace2024/displace2024_baseline_updated/blob/main/speaker_diarization/local/diarization/utils/scoring/wer_report.pl"

    %% Toolchain & Utilities
    subgraph "Tools & Utilities"
        direction TB
        Tools["Toolchain Installation"]:::kaldi
        Utils["Shared Utilities"]:::python
        Reqs["Requirements Files"]:::docs
    end

    Tools -->|provides Kaldi, dscore| FEat
    Tools -->|provides Kaldi| XVec
    Tools -->|provides dscore| RTTM
    Utils --> FEat
    Utils --> SC
    Utils --> RTTM
    click Tools "https://github.com/displace2024/displace2024_baseline_updated/blob/main/tools/install_kaldi.sh"
    click Tools "https://github.com/displace2024/displace2024_baseline_updated/blob/main/tools/install_dscore.sh"
    click Utils "https://github.com/displace2024/displace2024_baseline_updated/blob/main/utils/clustering.py"
    click Utils "https://github.com/displace2024/displace2024_baseline_updated/blob/main/utils/rttm_speaker_id.py"
    click Reqs "https://github.com/displace2024/displace2024_baseline_updated/blob/main/requirements/core.txt"
    click Reqs "https://github.com/displace2024/displace2024_baseline_updated/blob/main/requirements/sad.txt"

    %% Styles
    classDef python fill:#c8e6c9,stroke:#2e7d32,color:#1b5e20
    classDef kaldi fill:#ffe0b2,stroke:#ef6c00,color:#e65100
    classDef external fill:#bbdefb,stroke:#1565c0,color:#0d47a1
    classDef docs fill:#e0e0e0,stroke:#616161,color:#212121
```
