How maximyudayev/realtime-st-gcn works
Project Type and Purpose
• The project implements variants of ST-GCN (including RT-ST-GCN) for skeleton-based action recognition in real time.
• Its purpose is twofold: (a) to support continuous, real-time inference (with a lightweight model for embedded devices) and (b) to enable efficient, distributed training (using PyTorch’s Distributed Data Parallel and a novel gradient accumulation strategy for long, different-length sequences).
Key Components Derived from the File Structure
a. Data Handling and Preprocessing
• The “data/” directory stores multiple datasets (e.g., IMU-FOG-IT, PKU-MMD variants, and skeleton definitions).
• The “data_prep/” module (dataset.py, prep.py) abstracts the conversion of raw or directory-formatted data into unified 5D tensors along with corresponding labels.
• These components are responsible for converting different dataset formats into a standard input (N, C, L, V, M) for the models.
b. Configuration Management
• The “config/” directory contains JSON files that determine model architectures, optimizer settings, training parameters (e.g., segment sizes, batch sizes) and other execution parameters.
• This separation allows users to flexibly change model setups without modifying the underlying code.
c. Model Repository (Core Processing)
• The “models/” folder contains various implementations (e.g., aagcn, costgcn, msgcn, mstcn, rtstgcn, shiftgcn, shiftgcn_plus) and supporting utilities.
• These modules encapsulate the neural network architectures, and utilities (within models/utils) provide shared functions like batchnorm, graph construction, and layernorm operations.
d. Training, Inference, and Processing
• The “processor.py” script and “main.py” serve as the entry points. They parse configurations, organize data flow between modules, and orchestrate training/evaluation.
• Notable is the implementation of out-of-core training and manual gradient accumulation over data chunks (to bypass memory limitations).
e. Distributed Training Support
• The integration of PyTorch’s Distributed Data Parallel (DDP) is central for multi-GPU training.
• The provided SLURM and VSC scripts in the “vsc/” directory indicate deployment on HPC resources.
f. Utility and Evaluation Modules
• “utils/” (including metrics, loss functions, and visualization) supports model evaluation and performance tracking (e.g., F1 scores, confusion matrices).
• “tools/” (such as get_data.sh) assist in data/model downloads and setup procedures.
Relationships and Data Flow
• The configuration (from “config/” JSON files) is parsed by “utils/config_parser.py” and fed into both the training routine and model initialization.
• The “processor.py” (or “main.py”) module coordinates data loading, chunking for gradient accumulation, and invokes the appropriate model from “models/”.
• During training, data flows from the DataLoader to the Model where the forward pass happens, and gradients are accumulated manually if sequence lengths exceed memory.
• PyTorch’s DDP is employed for synchronizing training across multiple GPUs.
• Post-training, evaluation routines (using “utils/metrics/”) and benchmarking scripts (in “vsc/”) analyze performance and latency.
Architectural Patterns and Design Principles
• Modularity: Clear separation of concerns is seen – configuration, data preprocessing, model definition, training, and evaluation are decoupled.
• Scalability: Use of DDP and gradient accumulation allows scaling to multi-GPU and handling of data that cannot fit into memory fully.
• Extensibility: With many model variants and JSON-configurable parameters, the system is designed to experiment easily with new architectures and datasets.