How hulianyuyy/corrnet works
Project Type & Purpose
• This is a research prototype for Continuous Sign Language Recognition (SLR) based on the CorrNet architecture described in the CVPR 2023 paper.
• Its primary function is to process video/image inputs (of sign language) through deep learning modules (e.g., ResNet, BiLSTM) to produce sign language predictions.
• It supports multiple datasets (e.g., PHOENIX2014, PHOENIX2014-T, CSL, CSL-Daily) and offers training, inference (including one-video testing), and a demo UI.
Analyzing the File Structure
• Top-Level Organization:
• README.md: Overall project description and instructions.
• configs: Contains different YAML configuration files for training/inference (e.g., for different datasets).
• dataset: Holds dataset loaders (e.g., dataloader_video.py).
• evaluation: Contains scripts and utilities for performance evaluation (WER calculations, sclite integration, groundtruth files).
• modules: The heart of the neural network – contains implementations of network modules (e.g., ResNet, BiLSTM, temporal convolution, sync_batchnorm), as well as loss criterions.
• preprocess: Contains scripts to preprocess various datasets (resizing, generating gloss dictionaries, splitting data, etc.).
• utils: Provides miscellaneous helper scripts (device management, decoding utility, optimizer, etc.).
• Others: demo.py, main.py, and test_one_video.py orchestrate different use cases (training, inference, demo deployment).
Main Components to be Illustrated
a. Data/Preprocessing Components:
• Data Sources: Raw dataset directories (PHOENIX, CSL, etc.) and configuration files reside in “configs” and “dataset”.
• Preprocessing Pipelines: Scripts in "preprocess" that handle image resizing, gloss dictionary creation, and set up data splits.
b. Model Components:
• Neural Network Backbone: The CorrNet model architecture primarily built using a ResNet backbone (located in modules/resnet.py) enhanced with CorrNet-specific blocks. Also includes modules like BiLSTM (modules/BiLSTM.py) and temporal convolution (modules/tconv.py).
• Sync Batch Normalization: A dedicated submodule directory (modules/sync_batchnorm) for multi-device batch norm functionality.
c. Training & Inference Control:
• Main script (main.py): Responsible for training orchestration, argument parsing (command line > config file priority), and dataset selection.
• Single Video Testing (test_one_video.py) and Demo (demo.py): These scripts handle inference with different input types (video file versus interactive demo).
d. Evaluation:
• Evaluation Tools: Evaluation scripts inside the “evaluation” directory that include Python-based WER computation, sclite integration, score calculation, and supporting shell scripts.
e. Utility & Configuration:
• Utility scripts: Located under “utils” to support decoding, device management, optimizer settings, etc.
Relationships and Data Flow
• Input Data Flow: Raw video/sequence inputs → Preprocessing scripts (in “preprocess”) → Preprocessed data (e.g., resized images, gloss dicts).
• Model Training/Inference: Preprocessed data feeds into the deep learning model (modules) via the main training script (main.py). The model’s internal structure (ResNet, BiLSTM, etc.) processes the input and outputs predictions.
• Evaluation Pipeline: Model predictions pass to evaluation modules that use either in-house Python scripts or external tools (ctcdecode, sclite) to compute performance metrics (e.g., WER).
• Optional Demo Component: The demo.py script ties the inference module with a web frontend for real-time demonstration.
Important Architectural Patterns and Design Principles
• Modularity: Separation of concerns with distinct directories for data handling, pre-processing, model definition, evaluation, and demo.
• Configuration-Driven Execution: Usage of YAML configuration files, which are overridden by command-line arguments, emphasizing flexibility.
• Reusability & Extensibility: The network modules are designed so that modifications (like switching the CorrNet block for different datasets) are easy.
• Integration with External Tools: Interfaces with ctcdecode for beam search and sclite for detailed evaluation.