# ngoctuhan/sign-language-recognition-with-self-learning-fusion-models architecture

> Architecture diagram of the GitHub repository ngoctuhan/sign-language-recognition-with-self-learning-fusion-models, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models
- Repository: https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models

## Overview

##### Project type and purpose
- This is a research‑grade, multi‑modal video & sensor classification system.  Its goal is to recognize sign‑language gestures by training two separate models (video & sensor) and then applying self‑learning fusion to boost accuracy on three datasets (MHAD, UTD‑MHAD, VSL).

##### File structure
- Top‑level directories:
- dataset/ : CSV splits for three datasets
- utils/ : preprocessing scripts (video → .npy, sensor → .npy, CSV creation)
- nets/  : model definitions (inception3D for video, inception1D for sensor, basenet abstractions)
- fusion/: fusion strategies (late fusion, manifold fusion, raw variants)
- training/: orchestration pipeline for generator‑based self‑learning
- Key scripts:
- train_video.py
- train_sensor.py
- training_generator.py
- valid_generator.py
- Configuration & orchestration:
- config.ini
- main.sh

##### README for additional insights
- Two separate streams (RGB video & sensor time series)
- Video model based on I3D (Inception3D)
- Sensor model derived from Inception1D/basenet
- A “self‑learning generator” that fuses one model’s predictions into a second model to improve final accuracy

a. Main components
1. Data Ingestion
- Raw video files
- Raw sensor logs (accelerometer, gyroscope)
2. Preprocessing Layer
- utils/videoprocess.py → video frames (.npy)
- utils/sensorprocess.py → sensor arrays (.npy)
- utils/make_csv.py → train/test CSV splits under dataset/
3. Data Management
- dataset/MHAD, UTD‑MHAD, VSL (CSV files)
- numpy‑based dataset arrays
4. Model Definition
- nets/inception3D.py → VideoClassifier
- nets/inception1D.py → SensorClassifier
- nets/basenet.py → Common abstractions
5. Training Pipelines
- train_video.py
- train_sensor.py
6. Fusion & Generator
- fusion/late_fusion.py
- fusion/m_fusion.py
- training/pipeline.py (orchestration)
- training_generator.py → Self‑learning generator training
- valid_generator.py → Generator validation
7. Checkpoints & Outputs
- Model weights (.h5 or checkpoint files)
- Final fused predictions
8. Configuration & Orchestration
- config.ini
- main.sh

b. Relationships & interactions
- Raw Data → Preprocessing → Numpy Arrays & CSV splits
- CSV splits + Numpy Arrays → train_video.py → Video Model Checkpoint
- CSV splits + Numpy Arrays → train_sensor.py → Sensor Model Checkpoint
- Video & Sensor Checkpoints → training_generator.py → Generator Model
- Generator Model + Preprocessed Video → valid_generator.py → evaluation metrics
- Alternately, fusion scripts (fusion/*.py) combine the two base models’ outputs for late or manifold fusion

c. Architectural patterns
- Two‑stream architecture (video + sensor)
- Self‑learning generator acting as a second‑stage model
- Modular preprocessing / data pipeline
- Separation of concerns: preprocessing, model definition, training, fusion

d. Key technologies & frameworks
- Python 3
- TensorFlow / Keras (model training & definition)
- NumPy, Pandas (data handling)
- scikit‑learn (metrics)
- Configuration: configparser (config.ini)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Ingestion Layer
    subgraph "Data Ingestion" 
        direction TB
        RawVideo["Raw Video Files"]:::data
        RawSensor["Raw Sensor Logs"]:::data
    end

    %% Preprocessing Layer
    subgraph "Preprocessing Layer"
        direction TB
        PreprocessVideo["Video Preprocessor"]:::preproc
        PreprocessSensor["Sensor Preprocessor"]:::preproc
        CSVGen["CSV Split Generator"]:::preproc
    end

    %% Data Management
    subgraph "Data Management"
        direction TB
        DatasetMHAD["MHAD CSVs"]:::data
        DatasetUTD["UTD‑MHAD CSVs"]:::data
        DatasetVSL["VSL CSVs"]:::data
    end

    %% Model Definition
    subgraph "Model Definition"
        direction TB
        BaseNet["Base Abstractions"]:::model
        VideoClassifier["Video Classifier (I3D)"]:::model
        SensorClassifier["Sensor Classifier (I1D)"]:::model
    end

    %% Training Pipelines
    subgraph "Training Pipelines"
        direction TB
        TrainVideo["train_video.py"]:::train
        TrainSensor["train_sensor.py"]:::train
    end

    %% Fusion & Generator
    subgraph "Fusion & Generator"
        direction TB
        LateFusion["Late Fusion"]:::fusion
        LateFusionRaw["Late Fusion Raw"]:::fusion
        MFusion["Manifold Fusion"]:::fusion
        MFusionRaw["Manifold Fusion Raw"]:::fusion
        GeneratorTrain["training_generator.py"]:::fusion
        GeneratorValidate["valid_generator.py"]:::fusion
        Orchestration["training/pipeline.py"]:::train
    end

    %% Customs
    subgraph "Custom Modules"
        direction TB
        Loss["Custom Loss Functions"]:::customs
        Metrics["Custom Metrics"]:::customs
        SeqUTD["Sequence UTD-MHAD"]:::customs
        SeqVideo["Sequence Video"]:::customs
    end

    %% Configuration & Orchestration
    subgraph "Configuration & Orchestration"
        direction TB
        ConfigIni["config.ini"]:::config
        MainSh["main.sh"]:::config
    end

    %% Outputs
    Checkpoints["Model Checkpoints"]:::output
    FusionOutput["Final Fused Predictions"]:::output

    %% Connections
    RawVideo --> PreprocessVideo
    RawSensor --> PreprocessSensor
    PreprocessVideo --> CSVGen
    PreprocessSensor --> CSVGen
    CSVGen --> DatasetMHAD
    CSVGen --> DatasetUTD
    CSVGen --> DatasetVSL

    DatasetMHAD --> TrainVideo
    DatasetUTD --> TrainVideo
    DatasetVSL --> TrainVideo
    DatasetMHAD --> TrainSensor
    DatasetUTD --> TrainSensor
    DatasetVSL --> TrainSensor

    BaseNet --> VideoClassifier
    BaseNet --> SensorClassifier
    VideoClassifier --> TrainVideo
    SensorClassifier --> TrainSensor

    TrainVideo --> Checkpoints
    TrainSensor --> Checkpoints

    Checkpoints --> LateFusion
    Checkpoints --> LateFusionRaw
    Checkpoints --> MFusion
    Checkpoints --> MFusionRaw

    Checkpoints --> GeneratorTrain
    GeneratorTrain --> GeneratorValidate
    PreprocessVideo --> GeneratorValidate
    GeneratorValidate --> FusionOutput

    ConfigIni --> Orchestration
    MainSh --> Orchestration
    Orchestration --> TrainVideo
    Orchestration --> TrainSensor
    Orchestration --> GeneratorTrain
    Orchestration --> GeneratorValidate

    Loss --> TrainVideo
    Loss --> TrainSensor
    Metrics --> TrainVideo
    Metrics --> TrainSensor
    SeqUTD --> TrainSensor
    SeqVideo --> TrainVideo

    %% Click Events
    click PreprocessVideo "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/utils/videoprocess.py"
    click PreprocessSensor "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/utils/sensorprocess.py"
    click CSVGen "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/utils/make_csv.py"
    click DatasetMHAD "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/dataset/MHAD/train.csv"
    click DatasetMHAD "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/dataset/MHAD/test.csv"
    click DatasetUTD "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/dataset/UTD-MHAD/train.csv"
    click DatasetUTD "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/dataset/UTD-MHAD/test.csv"
    click DatasetVSL "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/dataset/VSL/train.csv"
    click DatasetVSL "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/dataset/VSL/test.csv"
    click DatasetVSL "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/dataset/VSL/shuf_train.csv"
    click DatasetVSL "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/dataset/VSL/shuf_test.csv"
    click DatasetVSL "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/dataset/VSL/vsl_train.csv"
    click DatasetVSL "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/dataset/VSL/vsl_test.csv"
    click BaseNet "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/nets/basenet.py"
    click VideoClassifier "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/nets/inception3D.py"
    click SensorClassifier "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/nets/inception1D.py"
    click TrainVideo "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/train_video.py"
    click TrainSensor "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/train_sensor.py"
    click Orchestration "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/training/pipeline.py"
    click GeneratorTrain "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/training_generator.py"
    click GeneratorValidate "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/valid_generator.py"
    click LateFusion "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/fusion/late_fusion.py"
    click LateFusionRaw "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/fusion/late_fusion_raw.py"
    click MFusion "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/fusion/m_fusion.py"
    click MFusionRaw "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/fusion/m_fusion_raw.py"
    click ConfigIni "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/config.ini"
    click MainSh "https://github.com/ngoctuhan/sign-language-recognition-with-self-learning-fusion-models/blob/main/main.sh"

    %% Styles
    classDef data fill:#ACE7EF,stroke:#036B81,color:#000
    classDef preproc fill:#B3E6CA,stroke:#1A7745,color:#000
    classDef model fill:#D5F5E3,stroke:#28B463,color:#000
    classDef train fill:#FDEBD0,stroke:#CA6F1E,color:#000
    classDef fusion fill:#E8DAEF,stroke:#7D3C98,color:#000
    classDef config fill:#D5D8DC,stroke:#566573,color:#000
    classDef customs fill:#FADBD8,stroke:#C0392B,color:#000
    classDef output fill:#F2F3F4,stroke:#839192,color:#000
```
