# hulianyuyy/corrnet architecture

> Architecture diagram of the GitHub repository hulianyuyy/corrnet, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hulianyuyy/corrnet
- Repository: https://github.com/hulianyuyy/corrnet

## Overview

##### Project Type & Purpose
- This is a research prototype for Continuous Sign Language Recognition (SLR) based on the CorrNet architecture described in the CVPR 2023 paper.
- Its primary function is to process video/image inputs (of sign language) through deep learning modules (e.g., ResNet, BiLSTM) to produce sign language predictions.
- It supports multiple datasets (e.g., PHOENIX2014, PHOENIX2014-T, CSL, CSL-Daily) and offers training, inference (including one-video testing), and a demo UI.

##### Analyzing the File Structure
- Top-Level Organization:
- README.md: Overall project description and instructions.
- configs: Contains different YAML configuration files for training/inference (e.g., for different datasets).
- dataset: Holds dataset loaders (e.g., dataloader_video.py).
- evaluation: Contains scripts and utilities for performance evaluation (WER calculations, sclite integration, groundtruth files).
- modules: The heart of the neural network – contains implementations of network modules (e.g., ResNet, BiLSTM, temporal convolution, sync_batchnorm), as well as loss criterions.
- preprocess: Contains scripts to preprocess various datasets (resizing, generating gloss dictionaries, splitting data, etc.).
- utils: Provides miscellaneous helper scripts (device management, decoding utility, optimizer, etc.).
- Others: demo.py, main.py, and test_one_video.py orchestrate different use cases (training, inference, demo deployment).

##### Main Components to be Illustrated
a. Data/Preprocessing Components:
- Data Sources: Raw dataset directories (PHOENIX, CSL, etc.) and configuration files reside in “configs” and “dataset”.
- Preprocessing Pipelines: Scripts in "preprocess" that handle image resizing, gloss dictionary creation, and set up data splits.
b. Model Components:
- Neural Network Backbone: The CorrNet model architecture primarily built using a ResNet backbone (located in modules/resnet.py) enhanced with CorrNet-specific blocks. Also includes modules like BiLSTM (modules/BiLSTM.py) and temporal convolution (modules/tconv.py).
- Sync Batch Normalization: A dedicated submodule directory (modules/sync_batchnorm) for multi-device batch norm functionality.
c. Training & Inference Control:
- Main script (main.py): Responsible for training orchestration, argument parsing (command line > config file priority), and dataset selection.
- Single Video Testing (test_one_video.py) and Demo (demo.py): These scripts handle inference with different input types (video file versus interactive demo).
d. Evaluation:
- Evaluation Tools: Evaluation scripts inside the “evaluation” directory that include Python-based WER computation, sclite integration, score calculation, and supporting shell scripts.
e. Utility & Configuration:
- Utility scripts: Located under “utils” to support decoding, device management, optimizer settings, etc.

##### Relationships and Data Flow
- Input Data Flow: Raw video/sequence inputs → Preprocessing scripts (in “preprocess”) → Preprocessed data (e.g., resized images, gloss dicts).
- Model Training/Inference: Preprocessed data feeds into the deep learning model (modules) via the main training script (main.py). The model’s internal structure (ResNet, BiLSTM, etc.) processes the input and outputs predictions.
- Evaluation Pipeline: Model predictions pass to evaluation modules that use either in-house Python scripts or external tools (ctcdecode, sclite) to compute performance metrics (e.g., WER).
- Optional Demo Component: The demo.py script ties the inference module with a web frontend for real-time demonstration.

##### Important Architectural Patterns and Design Principles
- Modularity: Separation of concerns with distinct directories for data handling, pre-processing, model definition, evaluation, and demo.
- Configuration-Driven Execution: Usage of YAML configuration files, which are overridden by command-line arguments, emphasizing flexibility.
- Reusability & Extensibility: The network modules are designed so that modifications (like switching the CorrNet block for different datasets) are easy.
- Integration with External Tools: Interfaces with ctcdecode for beam search and sclite for detailed evaluation.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data & Preprocessing Subgraph
    subgraph "Data & Preprocessing"
        cfg["Configuration Files (configs)"]:::data
        d1["Dataset Loader (dataset/dataloader_video.py)"]:::data
        preprocess["Preprocessing Pipeline (preprocess)"]:::data
    end

    %% Model Architecture Subgraph
    subgraph "Model Architecture"
        resnet["ResNet Backbone (modules/resnet.py)"]:::model
        bilstm["BiLSTM Module (modules/BiLSTM.py)"]:::model
        tconv["Temporal Convolution (modules/tconv.py)"]:::model
        syncbn["Sync Batch Normalization (modules/sync_batchnorm)"]:::model
    end

    %% Training & Inference Control Subgraph
    subgraph "Training & Inference Control"
        train["Main Training (main.py)"]:::control
        test["Single Video Inference (test_one_video.py)"]:::control
        demo["Demo UI (demo.py)"]:::control
    end

    %% Evaluation Subgraph
    subgraph "Evaluation"
        eval["Evaluation Tools (evaluation)"]:::evaluation
    end

    %% Utility Scripts Subgraph
    subgraph "Utilities"
        utils["Utility Scripts (utils)"]:::utility
    end

    %% Connections between components
    cfg --> d1
    d1 --> preprocess
    preprocess --> resnet
    preprocess --> bilstm
    preprocess --> tconv
    preprocess --> syncbn

    resnet --> train
    bilstm --> train
    tconv --> train
    syncbn --> train
    
    train --> test
    train --> demo
    demo --> eval
    test --> eval

    utils --- train
    utils --- resnet
    utils --- preprocess
    cfg --> train

    %% Click Events
    click cfg "https://github.com/hulianyuyy/corrnet/tree/main/configs"
    click d1 "https://github.com/hulianyuyy/corrnet/blob/main/dataset/dataloader_video.py"
    click preprocess "https://github.com/hulianyuyy/corrnet/tree/main/preprocess"
    click resnet "https://github.com/hulianyuyy/corrnet/blob/main/modules/resnet.py"
    click bilstm "https://github.com/hulianyuyy/corrnet/blob/main/modules/BiLSTM.py"
    click tconv "https://github.com/hulianyuyy/corrnet/blob/main/modules/tconv.py"
    click syncbn "https://github.com/hulianyuyy/corrnet/tree/main/modules/sync_batchnorm"
    click train "https://github.com/hulianyuyy/corrnet/blob/main/main.py"
    click test "https://github.com/hulianyuyy/corrnet/blob/main/test_one_video.py"
    click demo "https://github.com/hulianyuyy/corrnet/blob/main/demo.py"
    click eval "https://github.com/hulianyuyy/corrnet/tree/main/evaluation"
    click utils "https://github.com/hulianyuyy/corrnet/tree/main/utils"

    %% Styles
    classDef data fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef model fill:#ABEBC6,stroke:#145A32,stroke-width:2px;
    classDef control fill:#F9E79F,stroke:#B9770E,stroke-width:2px;
    classDef evaluation fill:#D2B4DE,stroke:#6C3483,stroke-width:2px;
    classDef utility fill:#E5E7E9,stroke:#616A6B,stroke-width:2px;
```
