# mpc001/auto_avsr architecture

> Architecture diagram of the GitHub repository mpc001/auto_avsr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mpc001/auto_avsr
- Repository: https://github.com/mpc001/auto_avsr

## Overview

##### Project type and purpose
- This is a full-stack, end-to-end machine-learning framework for audio-visual speech recognition (AVSR), with a strong emphasis on lip-reading (VSR) and audio speech recognition (ASR).
- It encompasses data preparation (preprocessing raw video/audio), feature extraction (SPM, transforms), model definition (ESPNet-based front-end, encoder, decoder, CTC/attention losses, beam search), training orchestration (PyTorch Lightning), checkpointing, logging, and evaluation.

##### Core components (to appear as primary “blocks”
A. Data Sources
- Raw video files, raw audio files (user dataset)
- Pretrained SPM model & units (spm/unigram…)
B. Data Preparation (preparation/)
- preprocessing scripts (preprocess_lrs2lrs3.py, preprocess_vox2.py)
- face detection & cropping (mediapipe, retinaface under detectors/)
- data merging / length filtering (merge.py, limit_length.py)
- transforms & utilities
C. Tokenization (spm/)
- spm_train.py → trains SentencePiece model
- spm_encode.py → performs subword encoding on transcripts
D. Data Module (datamodule/)
- PyTorch Lightning DataModule (data_module.py)
- Dataset class (av_dataset.py)
- waveform/video transforms (transforms.py)
E. Model Definition (espnet/nets/)
- Front-end: 2D/1D ResNet (frontend/)
- Encoder: Conformer, Transformer (encoder/)
- Decoder: Transformer decoder, CTC prefix scoring, beam search (decoder/, batch_beam_search.py)
- Loss functions & utilities (ctc.py, label_smoothing_loss.py, nets_utils.py)
- Scorer interface (scorer_interface.py, scorers/)
F. Training Orchestration
- train.py / lightning.py → configures PyTorch Lightning Trainer
- average_checkpoints.py / cosine.py → checkpoint averaging & LR schedules
- Logging & experiment tracking (WandB, TensorBoard via Lightning)
G. Evaluation & Inference
- eval.py → loads checkpoint, runs recognition on test set
- asr_infer.py under preparation/ → script for inference pipelines
- tutorials (notebooks) demonstrate feature extraction, mouth cropping, inference

##### Relationships & data flow
1. Raw data → Preparation pipelines
→ cropped lip frames + audio segments
2. Prepared data → SentencePiece encoding → text labels
3. Encoded data + audio/video → DataModule → batched PyTorch Tensors
4. Batches → Model (frontend → encoder → decoder/CTC loss)
5. Model outputs → Scorers & beam search → transcript hypotheses
6. Hypotheses → Evaluation metrics (WER)
7. During training → checkpoints & logs → average_checkpoints / cosine scheduling → resumed training or final model

##### Architectural patterns & design principles
- Modular separation: data prep, tokenization, data loading, model definition, training harness, evaluation.
- Reuse of ESPNet submodules for maintainability.
- Pluggable scorers/beam search via interface abstraction.
- Lightning for decoupling training loop from model logic.
- Script-based orchestration for reproducibility.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources (green)
    subgraph "Data Sources" 
        style DataSources fill:#e6ffe6,stroke:#333,stroke-width:1px
        RawVideo[(Raw Video Files)]:::data
        RawAudio[(Raw Audio Files)]:::data
        PretrainedSPM[(Pretrained SPM Model)]:::data
    end

    %% Data Preparation (blue)
    subgraph "Data Preparation (preparation/)" 
        style Prep fill:#e6f0ff,stroke:#333,stroke-width:1px
        Prep1["preprocess_lrs2lrs3.py"]:::processing
        Prep2["preprocess_vox2.py"]:::processing
        Prep3["merge.py"]:::processing
        Prep4["limit_length.py"]:::processing
        Prep5["transforms.py"]:::processing
        Prep6["utils.py"]:::processing
        subgraph "detectors/"
            style Det fill:#e6f0ff,stroke:#333,stroke-width:1px
            Mediapipe["mediapipe"]:::processing
            Retinaface["retinaface"]:::processing
        end
    end

    %% Tokenization (blue)
    subgraph "Tokenization (spm/)" 
        style SPM fill:#e6f0ff,stroke:#333,stroke-width:1px
        SPMTrain["spm_train.py"]:::processing
        SPMEncode["spm_encode.py"]:::processing
        UnigramModel[(unigram5000.model)]:::data
        UnigramUnits[(unigram5000_units.txt)]:::data
    end

    %% DataModule (blue)
    subgraph "DataModule (datamodule/)" 
        style DM fill:#e6f0ff,stroke:#333,stroke-width:1px
        DataModule["data_module.py"]:::processing
        AVDataset["av_dataset.py"]:::processing
        DMTransforms["transforms.py"]:::processing
        BabbleNoise[(babble_noise.wav)]:::data
    end

    %% Model Definition (orange)
    subgraph "Model Definition (espnet/nets/)" 
        style Model fill:#ffe6cc,stroke:#333,stroke-width:1px
        subgraph Frontend
            style FE fill:#ffe6cc,stroke:#333
            ResNet2D["resnet.py"]:::model
            ResNet1D["resnet1d.py"]:::model
        end
        subgraph Encoder
            style Enc fill:#ffe6cc,stroke:#333
            Conformer["conformer_encoder.py"]:::model
        end
        subgraph Decoder
            style Dec fill:#ffe6cc,stroke:#333
            TransformerDec["transformer_decoder.py"]:::model
            BeamSearch["beam_search.py"]:::model
            BatchBeam["batch_beam_search.py"]:::model
            CTCPrefix["ctc_prefix_score.py"]:::model
        end
        subgraph LossAndUtils
            style LU fill:#ffe6cc,stroke:#333
            CTC["ctc.py"]:::model
            LabelSmooth["label_smoothing_loss.py"]:::model
            NetUtils["nets_utils.py"]:::model
        end
        ScorerInterface["scorer_interface.py"]:::model
        ScorersDir["scorers/"]:::model
    end

    %% Orchestration (purple)
    subgraph "Training Orchestration" 
        style Orch fill:#f0e6ff,stroke:#333,stroke-width:1px
        TrainScript["train.py"]:::orchestration
        LightningConf["lightning.py"]:::orchestration
        AvgCheckpoints["average_checkpoints.py"]:::orchestration
        CosineLR["cosine.py"]:::orchestration
        Trainer[(PyTorch Lightning Trainer)]:::external
        GPUCluster((GPU Cluster)):::external
        WandB((WandB / TensorBoard)):::external
    end

    %% Evaluation & Inference (red)
    subgraph "Evaluation & Inference" 
        style Eval fill:#ffe6e6,stroke:#333,stroke-width:1px
        EvalScript["eval.py"]:::evaluation
        ASRInfer["asr_infer.py"]:::evaluation
        NotebookFE[(feature_extraction.ipynb)]:::evaluation
        NotebookCrop[(mouth_cropping.ipynb)]:::evaluation
        NotebookInfer[(inference.ipynb)]:::evaluation
    end

    %% External Libraries
    subgraph "External Dependencies" 
        style Ext fill:#ffffff,stroke:#777,stroke-width:1px
        PyTorch((PyTorch)):::external
        ESPNet((ESPNet)):::external
        SentencePiece((SentencePiece)):::external
    end

    %% Connections
    RawVideo -->|feeds| Prep1
    RawAudio -->|feeds| Prep1
    RawVideo --> Prep2
    RawAudio --> Prep2
    RawVideo --> Prep3
    RawAudio --> Prep4
    Prep1 --> Prep5
    Prep2 --> Prep5
    Prep3 --> Prep5
    Prep4 --> Prep5
    Prep5 --> Prep6
    Prep6 --> DM
    MediaPipe -.-> Prep5
    Retinaface -.-> Prep5

    Prep6 --> SPMTrain
    SPMTrain --> UnigramModel
    SPMTrain --> UnigramUnits
    UnigramModel --> SPMEncode
    UnigramUnits --> SPMEncode
    SPMEncode --> DataModule
    DataModule --> DataModule
    DataModule --> AVDataset
    DataModule --> DMTransforms

    DMTransforms -->|batches| DataModule
    DataModule -->|tensor batches| ResNet2D
    DataModule --> ResNet1D

    ResNet2D --> Conformer
    ResNet1D --> Conformer
    Conformer --> TransformerDec
    TransformerDec -->|decodes| CTC
    CTC -->|loss calc| TransformerDec
    TransformerDec --> BeamSearch
    BeamSearch -->|scoring| ScorerInterface
    ScorerInterface --> ScorersDir
    BeamSearch -->|outputs| EvalScript

    TrainScript --> Trainer
    LightningConf --> Trainer
    Trainer -->|runs| ResNet2D
    Trainer -->|runs| Conformer
    Trainer -->|runs| TransformerDec
    Trainer -->|logs| WandB
    Trainer -->|saves| AvgCheckpoints
    AvgCheckpoints -->|updates| Trainer
    CosineLR -->|schedules| Trainer
    GPUCluster --> Trainer

    EvalScript -->|evaluates| WandB
    ASRInfer -->|inference| TransformerDec
    NotebookFE -.-> Prep5
    NotebookCrop -.-> Prep5
    NotebookInfer -.-> ASRInfer

    %% Click Events
    click Prep1 "https://github.com/mpc001/auto_avsr/blob/main/preparation/preprocess_lrs2lrs3.py"
    click Prep2 "https://github.com/mpc001/auto_avsr/blob/main/preparation/preprocess_vox2.py"
    click Prep3 "https://github.com/mpc001/auto_avsr/blob/main/preparation/merge.py"
    click Prep4 "https://github.com/mpc001/auto_avsr/blob/main/preparation/limit_length.py"
    click Prep5 "https://github.com/mpc001/auto_avsr/blob/main/preparation/transforms.py"
    click Prep6 "https://github.com/mpc001/auto_avsr/blob/main/preparation/utils.py"
    click Mediapipe "https://github.com/mpc001/auto_avsr/tree/main/preparation/detectors/mediapipe/"
    click Retinaface "https://github.com/mpc001/auto_avsr/tree/main/preparation/detectors/retinaface/"
    click SPMTrain "https://github.com/mpc001/auto_avsr/blob/main/spm/spm_train.py"
    click SPMEncode "https://github.com/mpc001/auto_avsr/blob/main/spm/spm_encode.py"
    click UnigramModel "https://github.com/mpc001/auto_avsr/blob/main/spm/unigram/unigram5000.model"
    click UnigramUnits "https://github.com/mpc001/auto_avsr/blob/main/spm/unigram/unigram5000_units.txt"
    click DataModule "https://github.com/mpc001/auto_avsr/blob/main/datamodule/data_module.py"
    click AVDataset "https://github.com/mpc001/auto_avsr/blob/main/datamodule/av_dataset.py"
    click DMTransforms "https://github.com/mpc001/auto_avsr/blob/main/datamodule/transforms.py"
    click TrainScript "https://github.com/mpc001/auto_avsr/blob/main/train.py"
    click LightningConf "https://github.com/mpc001/auto_avsr/blob/main/lightning.py"
    click AvgCheckpoints "https://github.com/mpc001/auto_avsr/blob/main/average_checkpoints.py"
    click CosineLR "https://github.com/mpc001/auto_avsr/blob/main/cosine.py"
    click EvalScript "https://github.com/mpc001/auto_avsr/blob/main/eval.py"
    click ASRInfer "https://github.com/mpc001/auto_avsr/blob/main/preparation/asr_infer.py"
    click NotebookFE "https://github.com/mpc001/auto_avsr/blob/main/tutorials/feature_extraction.ipynb"
    click NotebookCrop "https://github.com/mpc001/auto_avsr/blob/main/tutorials/mouth_cropping.ipynb"
    click NotebookInfer "https://github.com/mpc001/auto_avsr/blob/main/tutorials/inference.ipynb"
    click Conformer "https://github.com/mpc001/auto_avsr/blob/main/espnet/nets/pytorch_backend/encoder/conformer_encoder.py"
    click TransformerDec "https://github.com/mpc001/auto_avsr/blob/main/espnet/nets/pytorch_backend/decoder/transformer_decoder.py"
    click BeamSearch "https://github.com/mpc001/auto_avsr/blob/main/espnet/nets/beam_search.py"
    click BatchBeam "https://github.com/mpc001/auto_avsr/blob/main/espnet/nets/batch_beam_search.py"
    click CTCPrefix "https://github.com/mpc001/auto_avsr/blob/main/espnet/nets/ctc_prefix_score.py"
    click CTC "https://github.com/mpc001/auto_avsr/blob/main/espnet/nets/pytorch_backend/ctc.py"
    click LabelSmooth "https://github.com/mpc001/auto_avsr/blob/main/espnet/nets/pytorch_backend/transformer/label_smoothing_loss.py"
    click NetUtils "https://github.com/mpc001/auto_avsr/blob/main/espnet/nets/pytorch_backend/nets_utils.py"
    click ScorerInterface "https://github.com/mpc001/auto_avsr/blob/main/espnet/nets/scorer_interface.py"

    %% Styles
    classDef data fill:#e6ffe6,stroke:#333,stroke-width:1px
    classDef processing fill:#e6f0ff,stroke:#333,stroke-width:1px
    classDef model fill:#ffe6cc,stroke:#333,stroke-width:1px
    classDef orchestration fill:#f0e6ff,stroke:#333,stroke-width:1px
    classDef evaluation fill:#ffe6e6,stroke:#333,stroke-width:1px
    classDef external fill:#ffffff,stroke:#999,stroke-dasharray: 5 5
```
