How ngoctuhan/sign-language-recognition-with-self-learning-fusion-models works
Project type and purpose
• This is a research‑grade, multi‑modal video & sensor classification system. Its goal is to recognize sign‑language gestures by training two separate models (video & sensor) and then applying self‑learning fusion to boost accuracy on three datasets (MHAD, UTD‑MHAD, VSL).
File structure
• Top‑level directories:
• dataset/ : CSV splits for three datasets
• utils/ : preprocessing scripts (video → .npy, sensor → .npy, CSV creation)
• nets/ : model definitions (inception3D for video, inception1D for sensor, basenet abstractions)
• fusion/: fusion strategies (late fusion, manifold fusion, raw variants)
• training/: orchestration pipeline for generator‑based self‑learning
• Key scripts:
• train_video.py
• train_sensor.py
• training_generator.py
• valid_generator.py
• Configuration & orchestration:
• config.ini
• main.sh
README for additional insights
• Two separate streams (RGB video & sensor time series)
• Video model based on I3D (Inception3D)
• Sensor model derived from Inception1D/basenet
• A “self‑learning generator” that fuses one model’s predictions into a second model to improve final accuracy
a. Main components
1. Data Ingestion
• Raw video files
• Raw sensor logs (accelerometer, gyroscope)
2. Preprocessing Layer
• utils/videoprocess.py → video frames (.npy)
• utils/sensorprocess.py → sensor arrays (.npy)
• utils/make_csv.py → train/test CSV splits under dataset/
3. Data Management
• dataset/MHAD, UTD‑MHAD, VSL (CSV files)
• numpy‑based dataset arrays
4. Model Definition
• nets/inception3D.py → VideoClassifier
• nets/inception1D.py → SensorClassifier
• nets/basenet.py → Common abstractions
5. Training Pipelines
• train_video.py
• train_sensor.py
6. Fusion & Generator
• fusion/late_fusion.py
• fusion/m_fusion.py
• training/pipeline.py (orchestration)
• training_generator.py → Self‑learning generator training
• valid_generator.py → Generator validation
7. Checkpoints & Outputs
• Model weights (.h5 or checkpoint files)
• Final fused predictions
8. Configuration & Orchestration
• config.ini
• main.sh
b. Relationships & interactions
• Raw Data → Preprocessing → Numpy Arrays & CSV splits
• CSV splits + Numpy Arrays → train_video.py → Video Model Checkpoint
• CSV splits + Numpy Arrays → train_sensor.py → Sensor Model Checkpoint
• Video & Sensor Checkpoints → training_generator.py → Generator Model
• Generator Model + Preprocessed Video → valid_generator.py → evaluation metrics
• Alternately, fusion scripts (fusion/*.py) combine the two base models’ outputs for late or manifold fusion
c. Architectural patterns
• Two‑stream architecture (video + sensor)
• Self‑learning generator acting as a second‑stage model
• Modular preprocessing / data pipeline
• Separation of concerns: preprocessing, model definition, training, fusion
d. Key technologies & frameworks
• Python 3
• TensorFlow / Keras (model training & definition)
• NumPy, Pandas (data handling)
• scikit‑learn (metrics)
• Configuration: configparser (config.ini)