How mpc001/auto_avsr works
Project type and purpose
• This is a full-stack, end-to-end machine-learning framework for audio-visual speech recognition (AVSR), with a strong emphasis on lip-reading (VSR) and audio speech recognition (ASR).
• It encompasses data preparation (preprocessing raw video/audio), feature extraction (SPM, transforms), model definition (ESPNet-based front-end, encoder, decoder, CTC/attention losses, beam search), training orchestration (PyTorch Lightning), checkpointing, logging, and evaluation.
Core components (to appear as primary “blocks”
A. Data Sources
• Raw video files, raw audio files (user dataset)
• Pretrained SPM model & units (spm/unigram…)
B. Data Preparation (preparation/)
• preprocessing scripts (preprocess_lrs2lrs3.py, preprocess_vox2.py)
• face detection & cropping (mediapipe, retinaface under detectors/)
• data merging / length filtering (merge.py, limit_length.py)
• transforms & utilities
C. Tokenization (spm/)
• spm_train.py → trains SentencePiece model
• spm_encode.py → performs subword encoding on transcripts
D. Data Module (datamodule/)
• PyTorch Lightning DataModule (data_module.py)
• Dataset class (av_dataset.py)
• waveform/video transforms (transforms.py)
E. Model Definition (espnet/nets/)
• Front-end: 2D/1D ResNet (frontend/)
• Encoder: Conformer, Transformer (encoder/)
• Decoder: Transformer decoder, CTC prefix scoring, beam search (decoder/, batch_beam_search.py)
• Loss functions & utilities (ctc.py, label_smoothing_loss.py, nets_utils.py)
• Scorer interface (scorer_interface.py, scorers/)
F. Training Orchestration
• train.py / lightning.py → configures PyTorch Lightning Trainer
• average_checkpoints.py / cosine.py → checkpoint averaging & LR schedules
• Logging & experiment tracking (WandB, TensorBoard via Lightning)
G. Evaluation & Inference
• eval.py → loads checkpoint, runs recognition on test set
• asr_infer.py under preparation/ → script for inference pipelines
• tutorials (notebooks) demonstrate feature extraction, mouth cropping, inference
Relationships & data flow
1. Raw data → Preparation pipelines
→ cropped lip frames + audio segments
2. Prepared data → SentencePiece encoding → text labels
3. Encoded data + audio/video → DataModule → batched PyTorch Tensors
4. Batches → Model (frontend → encoder → decoder/CTC loss)
5. Model outputs → Scorers & beam search → transcript hypotheses
6. Hypotheses → Evaluation metrics (WER)
7. During training → checkpoints & logs → average_checkpoints / cosine scheduling → resumed training or final model
Architectural patterns & design principles
• Modular separation: data prep, tokenization, data loading, model definition, training harness, evaluation.
• Reuse of ESPNet submodules for maintainability.
• Pluggable scorers/beam search via interface abstraction.
• Lightning for decoupling training loop from model logic.
• Script-based orchestration for reproducibility.