How y3162/shinoda-lab works
This repository is a collection of Shinoda Lab speech-research projects rather than one application. A central workflow prepares utterance/noise configurations, trains speech-enhancement models, then evaluates enhanced noisy speech with ASR; a separate ASR-error-modeling project trains a text-prediction model from recognition results. The map includes those distinct stages and the documented model families. Sampled source confirms enhancement loading/inference and parts of ASR inference/training; several database and evaluation links are inferred only where filenames explicitly describe the operation, so uncertain wiring is omitted.
Main entry points: observation_asr_results_fill_common.py Enhanced-audio ASR · ntp.py Text prediction training · database Dataset preparation · train.py SE training
Components 24
Audio data
- Speech corpora
- Noise corpora
- Dataset preparation
speech-enhancement/src/database - Noise configurations
speech-enhancement/src/database/03_2_insert_noise_configs_from_demand.py
Speech enhancement
- SE training
speech-enhancement/src/se_mamba/train.py - MP-SENet training
speech-enhancement/src/mp_senet/train.py - Mamba++ training
speech-enhancement/src/se_mamba_pp/train.py - Audio Mamba training
speech-enhancement/src/audio_mamba/train.py - Enhancement models
speech-enhancement/src/se_mamba/model - Enhancement API
speech-enhancement/src/api.py - Model checkpoints
ASR evaluation
- Evaluation runs
post-se-mixing/src/database/04_1_create_table_observation_eval_runs.py - Enhanced-audio ASR
post-se-mixing/src/database/observation_asr_results_fill_common.py - ASR model API
post-se-mixing/src/asr/api.py - ASR models
post-se-mixing/src/asr - ASR observations
post-se-mixing/src/database/05_1_create_table_observation_asr_results.py
Error modeling
- Noisy ASR inference
asr-error-modeling/src/database/04_2_noisy_inference.py - Recognition results
asr-error-modeling/src/database/04_1_create_table_asr_results.py - ASR pair dataset
asr-error-modeling/src/train/dataset.py - Text prediction training
asr-error-modeling/src/train/ntp.py - Transformer model
asr-error-modeling/src/model/transformer.py - Text tokenizers
asr-error-modeling/src/model/tokenizer.py
Other components
- Researcher
- Pretrained ASR
Connections 26
- Researcher → Dataset preparation: prepares data
- Speech corpora → Dataset preparation: supplies speech
- Noise corpora → Dataset preparation: supplies noise
- Dataset preparation → Noise configurations: creates configs
- Researcher → SE training: trains model
- SE training → Enhancement models: builds
- MP-SENet training → Model checkpoints: saves weights
- SE training → Model checkpoints: saves weights
- Mamba++ training → Model checkpoints: saves weights
- Audio Mamba training → Model checkpoints: saves weights
- Researcher → Evaluation runs: records run
- Evaluation runs → Enhanced-audio ASR: provides run
- Enhanced-audio ASR → Enhancement API: enhances audio
- Enhancement API → Model checkpoints: loads weights
- Enhanced-audio ASR → ASR model API: transcribes mixes
- ASR model API → ASR models: dispatches
- ASR models → Pretrained ASR: loads pretrained
- Enhanced-audio ASR → ASR observations: writes observations
- Noise configurations → Noisy ASR inference: configures noise
- Noisy ASR inference → Pretrained ASR: transcribes audio
- Noisy ASR inference → Recognition results: stores hypotheses
- Recognition results → ASR pair dataset: provides pairs
- Researcher → Text prediction training: trains predictor
- Text prediction training → ASR pair dataset: loads pairs
- Text prediction training → Text tokenizers: encodes text
- Text prediction training → Transformer model: optimizes