# ashigarg123/multimodal-speaker-verification architecture

> Architecture diagram of the GitHub repository ashigarg123/multimodal-speaker-verification, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ashigarg123/multimodal-speaker-verification
- Repository: https://github.com/ashigarg123/multimodal-speaker-verification

## Overview

This repository implements research workflows for multi-utterance speaker verification, including audio, prosody, and text models, and evaluates whether those signals remain useful on anonymized speech. The main path is configured data and trial preparation, model training or checkpoint loading, then trial-side embedding and scoring. A separate StreamVoiceAnon subsystem supports speech anonymization and its inference workflow. The graph reflects documented capabilities and indexed references; several implementation details—especially model variants and anonymization internals—were not sampled, so their links are limited to documented relationships.

Main entry points: [`scripts/train.py`](https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/scripts/train.py) (Training command), [`src/multimodal_asv/evaluation/trials.py`](https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/evaluation/trials.py) (Trial encoding and scoring), [`scripts/eval_trials.py`](https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/scripts/eval_trials.py) (Trial evaluation command), [`src/multimodal_asv/anonymization`](https://github.com/ashigarg123/multimodal-speaker-verification/tree/main/src/multimodal_asv/anonymization) (Anonymization workflow)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_experiment["Experiment workflow"]
  node_prep["Data preparation"]
  node_config["Experiment config<br/>[config.py]"]
  node_train_entry["Training command<br/>[train.py]"]
  node_training_data["Audio datasets<br/>[audio.py]"]
  node_trainer["Training loop<br/>[trainer.py]"]
  node_losses["Speaker losses<br/>[losses.py]"]
end

subgraph group_asv["Verification models"]
  node_registry["Model registry<br/>[registry.py]"]
  node_model_families["Aggregation models"]
  node_audio_encoder["Audio encoders<br/>[audio.py]"]
  node_text_encoder["LUAR text encoder<br/>[text.py]"]
  node_prosody["Prosody features<br/>[prosody.py]"]
  node_asset_resolver["Checkpoint assets<br/>[assets.py]"]
end

subgraph group_evaluation["Trial evaluation"]
  node_eval_entry["Trial evaluation command<br/>[eval_trials.py]"]
  node_trial_eval["Trial encoding and scoring<br/>[trials.py]"]
  node_eval_common["Evaluation metrics<br/>[common.py]"]
  node_text_eval["Text-only evaluation<br/>[text_only.py]"]
end

subgraph group_anonymization["Speech anonymization"]
  node_anon_workflow["Anonymization workflow"]
  node_anon_inference["StreamVoiceAnon inference<br/>[infer_arvc.py]"]
  node_anon_data["Anonymizer data loading"]
  node_anon_model["StreamVoiceAnon models"]
  node_anon_trainer["Anonymizer training<br/>[arvc_trainer.py]"]
end

node_researcher(("Researcher"))
node_corpora[("Speech and transcripts")]

node_researcher -->|"prepares data"| node_prep
node_corpora -->|"provides speech"| node_prep
node_researcher -->|"starts training"| node_train_entry
node_train_entry -->|"loads config"| node_config
node_train_entry -->|"builds datasets"| node_training_data
node_train_entry -->|"builds model"| node_registry
node_train_entry -->|"starts training"| node_trainer
node_training_data -->|"extracts features"| node_prosody
node_trainer -->|"uses loss"| node_losses
node_registry -->|"dispatches to"| node_model_families
node_model_families -.->|"encodes audio"| node_audio_encoder
node_model_families -.->|"encodes text"| node_text_encoder
node_text_encoder -->|"resolves checkpoint"| node_asset_resolver
node_researcher -->|"runs evaluation"| node_eval_entry
node_eval_entry -->|"loads config"| node_config
node_eval_entry -->|"evaluates trials"| node_trial_eval
node_trial_eval -->|"loads model"| node_registry
node_trial_eval -->|"extracts features"| node_prosody
node_trial_eval -->|"uses metrics"| node_eval_common
node_eval_entry -->|"supports text trials"| node_text_eval
node_corpora -->|"supplies speech"| node_anon_workflow
node_anon_workflow -->|"runs inference"| node_anon_inference
node_anon_inference -.->|"initializes models"| node_anon_model
node_anon_data -->|"feeds training"| node_anon_trainer
node_anon_workflow -.->|"prepares anonymizer data"| node_anon_data
node_anon_workflow -->|"evaluates anonymized speech"| node_trial_eval

click node_prep "https://github.com/ashigarg123/multimodal-speaker-verification/tree/main/scripts/data_prep"
click node_config "https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/config.py"
click node_train_entry "https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/scripts/train.py"
click node_training_data "https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/data/audio.py"
click node_trainer "https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/training/trainer.py"
click node_losses "https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/training/losses.py"
click node_registry "https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/registry.py"
click node_model_families "https://github.com/ashigarg123/multimodal-speaker-verification/tree/main/src/multimodal_asv/models"
click node_audio_encoder "https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/encoders/audio.py"
click node_text_encoder "https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/encoders/text.py"
click node_prosody "https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/utils/prosody.py"
click node_asset_resolver "https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/utils/assets.py"
click node_eval_entry "https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/scripts/eval_trials.py"
click node_trial_eval "https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/evaluation/trials.py"
click node_eval_common "https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/evaluation/common.py"
click node_text_eval "https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/evaluation/text_only.py"
click node_anon_workflow "https://github.com/ashigarg123/multimodal-speaker-verification/tree/main/src/multimodal_asv/anonymization"
click node_anon_inference "https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/anonymization/StreamVoiceAnon/evaluations/infer_arvc.py"
click node_anon_data "https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/anonymization/StreamVoiceAnon/dataloaders/hf_stream_dataloader.py"
click node_anon_model "https://github.com/ashigarg123/multimodal-speaker-verification/tree/main/src/multimodal_asv/anonymization/StreamVoiceAnon/modules"
click node_anon_trainer "https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/anonymization/StreamVoiceAnon/trainers/arvc_trainer.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_prep,node_config,node_train_entry,node_training_data,node_trainer,node_losses toneBlue
class node_registry,node_model_families,node_audio_encoder,node_text_encoder,node_prosody,node_asset_resolver,node_corpora toneAmber
class node_eval_entry,node_trial_eval,node_eval_common,node_text_eval toneMint
class node_anon_workflow,node_anon_inference,node_anon_data,node_anon_model,node_anon_trainer toneRose
class node_researcher toneIndigo
```

## Components

### Experiment workflow

- [Data preparation](https://github.com/ashigarg123/multimodal-speaker-verification/tree/main/scripts/data_prep): `scripts/data_prep`
- [Experiment config](https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/config.py): `src/multimodal_asv/config.py`
- [Training command](https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/scripts/train.py): `scripts/train.py`
- [Audio datasets](https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/data/audio.py): `src/multimodal_asv/data/audio.py`
- [Training loop](https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/training/trainer.py): `src/multimodal_asv/training/trainer.py`
- [Speaker losses](https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/training/losses.py): `src/multimodal_asv/training/losses.py`

### Verification models

- [Model registry](https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/registry.py): `src/multimodal_asv/registry.py`
- [Aggregation models](https://github.com/ashigarg123/multimodal-speaker-verification/tree/main/src/multimodal_asv/models): `src/multimodal_asv/models`
- [Audio encoders](https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/encoders/audio.py): `src/multimodal_asv/encoders/audio.py`
- [LUAR text encoder](https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/encoders/text.py): `src/multimodal_asv/encoders/text.py`
- [Prosody features](https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/utils/prosody.py): `src/multimodal_asv/utils/prosody.py`
- [Checkpoint assets](https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/utils/assets.py): `src/multimodal_asv/utils/assets.py`

### Trial evaluation

- [Trial evaluation command](https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/scripts/eval_trials.py): `scripts/eval_trials.py`
- [Trial encoding and scoring](https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/evaluation/trials.py): `src/multimodal_asv/evaluation/trials.py`
- [Evaluation metrics](https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/evaluation/common.py): `src/multimodal_asv/evaluation/common.py`
- [Text-only evaluation](https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/evaluation/text_only.py): `src/multimodal_asv/evaluation/text_only.py`

### Speech anonymization

- [Anonymization workflow](https://github.com/ashigarg123/multimodal-speaker-verification/tree/main/src/multimodal_asv/anonymization): `src/multimodal_asv/anonymization`
- [StreamVoiceAnon inference](https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/anonymization/StreamVoiceAnon/evaluations/infer_arvc.py): `src/multimodal_asv/anonymization/StreamVoiceAnon/evaluations/infer_arvc.py`
- [Anonymizer data loading](https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/anonymization/StreamVoiceAnon/dataloaders/hf_stream_dataloader.py): `src/multimodal_asv/anonymization/StreamVoiceAnon/dataloaders/hf_stream_dataloader.py`
- [StreamVoiceAnon models](https://github.com/ashigarg123/multimodal-speaker-verification/tree/main/src/multimodal_asv/anonymization/StreamVoiceAnon/modules): `src/multimodal_asv/anonymization/StreamVoiceAnon/modules`
- [Anonymizer training](https://github.com/ashigarg123/multimodal-speaker-verification/blob/main/src/multimodal_asv/anonymization/StreamVoiceAnon/trainers/arvc_trainer.py): `src/multimodal_asv/anonymization/StreamVoiceAnon/trainers/arvc_trainer.py`

### Other components

- **Researcher**
- **Speech and transcripts**

## Connections

- Researcher → Data preparation: prepares data (evidence: `README.md`)
- Speech and transcripts → Data preparation: provides speech (evidence: `README.md`)
- Researcher → Training command: starts training (evidence: `README.md`)
- Training command → Experiment config: loads config (evidence: `scripts/train.py`)
- Training command → Audio datasets: builds datasets (evidence: `scripts/train.py`)
- Training command → Model registry: builds model (evidence: `scripts/train.py`)
- Training command → Training loop: starts training (evidence: `scripts/train.py`)
- Audio datasets → Prosody features: extracts features (evidence: `src/multimodal_asv/data/audio.py`)
- Training loop → Speaker losses: uses loss (evidence: `src/multimodal_asv/training/trainer.py`)
- Model registry → Aggregation models: dispatches to (evidence: `src/multimodal_asv/registry.py`)
- Aggregation models → Audio encoders: encodes audio
- Aggregation models → LUAR text encoder: encodes text
- LUAR text encoder → Checkpoint assets: resolves checkpoint (evidence: `src/multimodal_asv/encoders/text.py`)
- Researcher → Trial evaluation command: runs evaluation (evidence: `README.md`)
- Trial evaluation command → Experiment config: loads config (evidence: `scripts/eval_trials.py`)
- Trial evaluation command → Trial encoding and scoring: evaluates trials (evidence: `scripts/eval_trials.py`)
- Trial encoding and scoring → Model registry: loads model (evidence: `src/multimodal_asv/evaluation/trials.py`)
- Trial encoding and scoring → Prosody features: extracts features (evidence: `src/multimodal_asv/evaluation/trials.py`)
- Trial encoding and scoring → Evaluation metrics: uses metrics (evidence: `src/multimodal_asv/evaluation/trials.py`)
- Trial evaluation command → Text-only evaluation: supports text trials
- Speech and transcripts → Anonymization workflow: supplies speech (evidence: `README.md`)
- Anonymization workflow → StreamVoiceAnon inference: runs inference (evidence: `src/multimodal_asv/anonymization/streamvoiceanon.py`)
- StreamVoiceAnon inference → StreamVoiceAnon models: initializes models
- Anonymizer data loading → Anonymizer training: feeds training (evidence: `src/multimodal_asv/anonymization/StreamVoiceAnon/trainers/arvc_trainer.py`)
- Anonymization workflow → Anonymizer data loading: prepares anonymizer data (evidence: `src/multimodal_asv/anonymization/StreamVoiceAnon/trainers/arvc_trainer.py`)
- Anonymization workflow → Trial encoding and scoring: evaluates anonymized speech (evidence: `README.md`)
