# maximyudayev/realtime-st-gcn architecture

> Architecture diagram of the GitHub repository maximyudayev/realtime-st-gcn, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/maximyudayev/realtime-st-gcn
- Repository: https://github.com/maximyudayev/realtime-st-gcn

## Overview

##### Project Type and Purpose
- The project implements variants of ST-GCN (including RT-ST-GCN) for skeleton-based action recognition in real time.
- Its purpose is twofold: (a) to support continuous, real-time inference (with a lightweight model for embedded devices) and (b) to enable efficient, distributed training (using PyTorch’s Distributed Data Parallel and a novel gradient accumulation strategy for long, different-length sequences).

##### Key Components Derived from the File Structure
a. Data Handling and Preprocessing
- The “data/” directory stores multiple datasets (e.g., IMU-FOG-IT, PKU-MMD variants, and skeleton definitions).
- The “data_prep/” module (dataset.py, prep.py) abstracts the conversion of raw or directory-formatted data into unified 5D tensors along with corresponding labels.
- These components are responsible for converting different dataset formats into a standard input (N, C, L, V, M) for the models.

b. Configuration Management
- The “config/” directory contains JSON files that determine model architectures, optimizer settings, training parameters (e.g., segment sizes, batch sizes) and other execution parameters.
- This separation allows users to flexibly change model setups without modifying the underlying code.

c. Model Repository (Core Processing)
- The “models/” folder contains various implementations (e.g., aagcn, costgcn, msgcn, mstcn, rtstgcn, shiftgcn, shiftgcn_plus) and supporting utilities.
- These modules encapsulate the neural network architectures, and utilities (within models/utils) provide shared functions like batchnorm, graph construction, and layernorm operations.

d. Training, Inference, and Processing
- The “processor.py” script and “main.py” serve as the entry points. They parse configurations, organize data flow between modules, and orchestrate training/evaluation.
- Notable is the implementation of out-of-core training and manual gradient accumulation over data chunks (to bypass memory limitations).

e. Distributed Training Support
- The integration of PyTorch’s Distributed Data Parallel (DDP) is central for multi-GPU training.
- The provided SLURM and VSC scripts in the “vsc/” directory indicate deployment on HPC resources.

f. Utility and Evaluation Modules
- “utils/” (including metrics, loss functions, and visualization) supports model evaluation and performance tracking (e.g., F1 scores, confusion matrices).
- “tools/” (such as get_data.sh) assist in data/model downloads and setup procedures.

##### Relationships and Data Flow
- The configuration (from “config/” JSON files) is parsed by “utils/config_parser.py” and fed into both the training routine and model initialization.
- The “processor.py” (or “main.py”) module coordinates data loading, chunking for gradient accumulation, and invokes the appropriate model from “models/”.
- During training, data flows from the DataLoader to the Model where the forward pass happens, and gradients are accumulated manually if sequence lengths exceed memory.
- PyTorch’s DDP is employed for synchronizing training across multiple GPUs.
- Post-training, evaluation routines (using “utils/metrics/”) and benchmarking scripts (in “vsc/”) analyze performance and latency.

##### Architectural Patterns and Design Principles
- Modularity: Clear separation of concerns is seen – configuration, data preprocessing, model definition, training, and evaluation are decoupled.
- Scalability: Use of DDP and gradient accumulation allows scaling to multi-GPU and handling of data that cannot fit into memory fully.
- Extensibility: With many model variants and JSON-configurable parameters, the system is designed to experiment easily with new architectures and datasets.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        DS["Raw Data (data/)"] 
        DLD["Dataset Loader (dataset.py)"]
        DPS["Data Prep (prep.py)"]
    end

    %% Configuration Management
    subgraph "Configuration Management"
        CONF["JSON Configurations (config/)"]
    end

    %% Training & Inference Orchestration
    subgraph "Training & Inference Orchestration"
        PROC["Processor (processor.py)"]
        MAIN["Main Entry (main.py)"]
    end

    %% Model Repository
    subgraph "Model Repository"
        MODEL_SEL["Model Selection"]
        AAGCN["AAGCN"]
        COSTGCN["COSTGCN"]
        MSGCN["MSGCN"]
        MSTCN["MSTCN"]
        RTSTGCN["RT-STGCN"]
        SHIFTGCN["ShiftGCN"]
        SHIFTGCNPlus["ShiftGCN+"]
        STGCN["ST-GCN"]
        MODELU["Utils (models/utils)"]
    end

    %% Distributed Training Support
    subgraph "Distributed Training Support"
        VSC["HPC & DDP (vsc/)"]
    end

    %% Utility & Evaluation Modules
    subgraph "Utility & Evaluation Modules"
        CP["Config Parser (utils/config_parser.py)"]
        LOSS["Loss Functions (utils/loss.py)"]
        METRICS["Metrics (utils/metrics)"]
        GETDATA["Get Data Script (tools/get_data.sh)"]
    end

    %% Data Flow Connections
    DS -->|"raw-to-loader"| DLD
    DLD -->|"to-tensors"| DPS

    CONF -->|"provides"| PROC
    DPS -->|"processed_data"| PROC
    PROC -->|"orchestrates"| MAIN

    MAIN -->|"model_selection"| MODEL_SEL
    MODEL_SEL -->|"invokes"| AAGCN
    MODEL_SEL -->|"invokes"| COSTGCN
    MODEL_SEL -->|"invokes"| MSGCN
    MODEL_SEL -->|"invokes"| MSTCN
    MODEL_SEL -->|"invokes"| RTSTGCN
    MODEL_SEL -->|"invokes"| SHIFTGCN
    MODEL_SEL -->|"invokes"| SHIFTGCNPlus
    MODEL_SEL -->|"invokes"| STGCN
    MODEL_SEL -->|"utilizes"| MODELU

    MAIN -->|"distributed_sync"| VSC
    MAIN -->|"evaluation_results"| METRICS

    CONF -->|"parsed_by"| CP
    CP -->|"setup"| GETDATA

    %% Styles
    classDef data fill:#F39C12,stroke:#000,stroke-width:1px;
    classDef config fill:#3498DB,stroke:#000,stroke-width:1px;
    classDef processing fill:#E67E22,stroke:#000,stroke-width:1px;
    classDef model fill:#2ECC71,stroke:#000,stroke-width:1px;
    classDef distributed fill:#9B59B6,stroke:#000,stroke-width:1px;
    classDef utility fill:#1ABC9C,stroke:#000,stroke-width:1px;

    class DS,DLD,DPS data;
    class CONF config;
    class PROC,MAIN processing;
    class MODEL_SEL,AAGCN,COSTGCN,MSGCN,MSTCN,RTSTGCN,SHIFTGCN,SHIFTGCNPlus,STGCN,MODELU model;
    class VSC distributed;
    class CP,LOSS,METRICS,GETDATA utility;

    %% Click Events
    click DS "https://github.com/maximyudayev/realtime-st-gcn/tree/main/data/"
    click DLD "https://github.com/maximyudayev/realtime-st-gcn/blob/main/data_prep/dataset.py"
    click DPS "https://github.com/maximyudayev/realtime-st-gcn/blob/main/data_prep/prep.py"
    click CONF "https://github.com/maximyudayev/realtime-st-gcn/tree/main/config/"
    click PROC "https://github.com/maximyudayev/realtime-st-gcn/blob/main/processor.py"
    click MAIN "https://github.com/maximyudayev/realtime-st-gcn/blob/main/main.py"
    click AAGCN "https://github.com/maximyudayev/realtime-st-gcn/tree/main/models/aagcn"
    click COSTGCN "https://github.com/maximyudayev/realtime-st-gcn/tree/main/models/costgcn"
    click MSGCN "https://github.com/maximyudayev/realtime-st-gcn/tree/main/models/msgcn"
    click MSTCN "https://github.com/maximyudayev/realtime-st-gcn/tree/main/models/mstcn"
    click RTSTGCN "https://github.com/maximyudayev/realtime-st-gcn/tree/main/models/rtstgcn"
    click SHIFTGCN "https://github.com/maximyudayev/realtime-st-gcn/tree/main/models/shiftgcn"
    click SHIFTGCNPlus "https://github.com/maximyudayev/realtime-st-gcn/tree/main/models/shiftgcn_plus"
    click STGCN "https://github.com/maximyudayev/realtime-st-gcn/tree/main/models/stgcn"
    click MODELU "https://github.com/maximyudayev/realtime-st-gcn/tree/main/models/utils"
    click VSC "https://github.com/maximyudayev/realtime-st-gcn/tree/main/vsc/"
    click CP "https://github.com/maximyudayev/realtime-st-gcn/blob/main/utils/config_parser.py"
    click LOSS "https://github.com/maximyudayev/realtime-st-gcn/blob/main/utils/loss.py"
    click METRICS "https://github.com/maximyudayev/realtime-st-gcn/tree/main/utils/metrics"
    click GETDATA "https://github.com/maximyudayev/realtime-st-gcn/blob/main/tools/get_data.sh"
```
