How ashigarg123/multimodal-speaker-verification works
This repository implements research workflows for multi-utterance speaker verification, including audio, prosody, and text models, and evaluates whether those signals remain useful on anonymized speech. The main path is configured data and trial preparation, model training or checkpoint loading, then trial-side embedding and scoring. A separate StreamVoiceAnon subsystem supports speech anonymization and its inference workflow. The graph reflects documented capabilities and indexed references; several implementation details—especially model variants and anonymization internals—were not sampled, so their links are limited to documented relationships.
Main entry points: train.py Training command · trials.py Trial encoding and scoring · eval_trials.py Trial evaluation command · anonymization Anonymization workflow
Components 23
Experiment workflow
- Data preparation
scripts/data_prep - Experiment config
src/multimodal_asv/config.py - Training command
scripts/train.py - Audio datasets
src/multimodal_asv/data/audio.py - Training loop
src/multimodal_asv/training/trainer.py - Speaker losses
src/multimodal_asv/training/losses.py
Verification models
- Model registry
src/multimodal_asv/registry.py - Aggregation models
src/multimodal_asv/models - Audio encoders
src/multimodal_asv/encoders/audio.py - LUAR text encoder
src/multimodal_asv/encoders/text.py - Prosody features
src/multimodal_asv/utils/prosody.py - Checkpoint assets
src/multimodal_asv/utils/assets.py
Trial evaluation
- Trial evaluation command
scripts/eval_trials.py - Trial encoding and scoring
src/multimodal_asv/evaluation/trials.py - Evaluation metrics
src/multimodal_asv/evaluation/common.py - Text-only evaluation
src/multimodal_asv/evaluation/text_only.py
Speech anonymization
- Anonymization workflow
src/multimodal_asv/anonymization - StreamVoiceAnon inference
src/multimodal_asv/anonymization/StreamVoiceAnon/evaluations/infer_arvc.py - Anonymizer data loading
src/multimodal_asv/anonymization/StreamVoiceAnon/dataloaders/hf_stream_dataloader.py - StreamVoiceAnon models
src/multimodal_asv/anonymization/StreamVoiceAnon/modules - Anonymizer training
src/multimodal_asv/anonymization/StreamVoiceAnon/trainers/arvc_trainer.py
Other components
- Researcher
- Speech and transcripts
Connections 26
- Researcher → Data preparation: prepares data
README.md - Speech and transcripts → Data preparation: provides speech
README.md - Researcher → Training command: starts training
README.md - Training command → Experiment config: loads config
scripts/train.py - Training command → Audio datasets: builds datasets
scripts/train.py - Training command → Model registry: builds model
scripts/train.py - Training command → Training loop: starts training
scripts/train.py - Audio datasets → Prosody features: extracts features
src/multimodal_asv/data/audio.py - Training loop → Speaker losses: uses loss
src/multimodal_asv/training/trainer.py - Model registry → Aggregation models: dispatches to
src/multimodal_asv/registry.py - Aggregation models → Audio encoders: encodes audio
- Aggregation models → LUAR text encoder: encodes text
- LUAR text encoder → Checkpoint assets: resolves checkpoint
src/multimodal_asv/encoders/text.py - Researcher → Trial evaluation command: runs evaluation
README.md - Trial evaluation command → Experiment config: loads config
scripts/eval_trials.py - Trial evaluation command → Trial encoding and scoring: evaluates trials
scripts/eval_trials.py - Trial encoding and scoring → Model registry: loads model
src/multimodal_asv/evaluation/trials.py - Trial encoding and scoring → Prosody features: extracts features
src/multimodal_asv/evaluation/trials.py - Trial encoding and scoring → Evaluation metrics: uses metrics
src/multimodal_asv/evaluation/trials.py - Trial evaluation command → Text-only evaluation: supports text trials
- Speech and transcripts → Anonymization workflow: supplies speech
README.md - Anonymization workflow → StreamVoiceAnon inference: runs inference
src/multimodal_asv/anonymization/streamvoiceanon.py - StreamVoiceAnon inference → StreamVoiceAnon models: initializes models
- Anonymizer data loading → Anonymizer training: feeds training
src/multimodal_asv/anonymization/StreamVoiceAnon/trainers/arvc_trainer.py - Anonymization workflow → Anonymizer data loading: prepares anonymizer data
src/multimodal_asv/anonymization/StreamVoiceAnon/trainers/arvc_trainer.py - Anonymization workflow → Trial encoding and scoring: evaluates anonymized speech
README.md