How sahityabiswas/gesture2sentence works
Purpose and Entry Points
Python-based sign-language recognition system for isolated signs, with optional sign-keyword-to-sentence generation. Primary operational entry points are training (Train_hierarchical.py), dataset-ID prediction (predict.py), raw-video prediction (predict_video.py), evaluation (evaluate.py), and interactive sentence demo (demo.py).
Keypoint Input and Feature Boundary
The core classifier consumes normalized keypoint sequences rather than video pixels. Raw videos pass through MediaPipe-based extraction and sequence preprocessing before normalization against persisted global statistics. Dataset-backed samples are loaded directly from stored keypoint data.
Grounding: video_keypoint_extractor.py, video_preprocess_utils.py, normalize.py, dataset.py, global_stats.pkl.
Hierarchical Recognition Training
Train_hierarchical.py orchestrates creation of label mappings, class-to-group clustering, group-model training, and one class submodel per group. The hierarchy reduces a large flat sign label space into group routing followed by within-group classification. Model structure and training dimensions are defined by the main model/config layer.
Grounding: Train_hierarchical.py, model.py, config.py, label_map.pkl, group_map.pkl.
Two-Stage Inference
Inference first scores sign groups, then evaluates class candidates within selected groups, combining/ranking predictions using configurable settings such as top-k and score calibration. Saved group/submodel checkpoints and mapping artifacts are an explicit boundary between training and serving; artifact location can be redirected through environment configuration.
Grounding: hierarchical_inference.py, inference_utils.py, inference_settings.json, predict.py, predict_video.py.
Evaluation and Artifact Lifecycle
Evaluation runs the trained hierarchy against dataset splits and supports parameter sweeps to select inference settings. Persisted data mappings, normalization statistics, group assignments, and omitted PyTorch checkpoints are required runtime dependencies; source control includes only small mapping/statistical artifacts visible in the tree.
Grounding: evaluate.py, dataset.csv, vid_class_FDMSE-ISL.pkl, vid_splits_FDMSE-ISL.pkl, class_map_FDMSE-ISL.csv.
Sign-to-Sentence Generation
A separate downstream pipeline predicts signs over a short video window, converts labels to keyword text, and invokes a T5 text generator to generate natural-language candidates. T5 training and direct keyword inference are isolated under wts_split; this stage depends on the recognition output but is not part of classifier training.
Grounding: pipeline.py, demo.py, wts_split/training.py, wts_split/inference.py, wts_split/dataset_word_gen.py.
Alternate Graph-Model Experiment
ctrgcn_workspace is a separate CTRGCN-based graph-recognition experiment that shares the keypoint representation concept but has independent dataset, graph definition, model, training, inference, and evaluation modules. It should be treated as an experimental parallel architecture, not as a dependency of the main hierarchical serving path.
Grounding: ctrgcn_workspace/graph.py, ctrgcn_workspace/model_ctrgcn.py, ctrgcn_workspace/train_ctrgcn.py, ctrgcn_workspace/evaluate_ctrgcn.py.
Runtime and External Dependencies
The runtime is Python with PyTorch for recognition models, NumPy/Pandas/scikit-learn for data and KMeans grouping, OpenCV and MediaPipe for video/keypoint extraction, and Hugging Face Transformers/T5 for language generation. CUDA-capable PyTorch is optional for acceleration; no web service, database, or deployment infrastructure is represented.
Main entry points: hierarchical_inference.py Two-stage inference · dataset.py Dataset loader · video_keypoint_extractor.py Keypoint extraction · normalize.py Sequence normalization
Components 19
Keypoint Data
Shared input representation and persisted dataset artifacts.
- Raw videos
- Keypoint extraction · Extracts and preprocesses video pose/keypoint sequences.
video_keypoint_extractor.py - Sequence normalization · Normalizes keypoint sequences with persisted global statistics.
normalize.py - Dataset loader · Loads stored keypoint samples and dataset splits.
dataset.py - Dataset and statistics · Dataset metadata, splits, class map, keypoints, and global statistics.
dataset.csv
Hierarchical Recognition
Primary sign classifier training, artifacts, and serving.
- Hierarchy training · Builds label/group mappings and trains routing and group-specific classifiers.
Train_hierarchical.py - Recognition model · Defines classifier structure and training dimensions.
model.py - Hierarchy artifacts · Label/group mappings and saved group/submodel checkpoints.
label_map.pkl - Two-stage inference · Routes through sign groups then ranks within-group class candidates.
hierarchical_inference.py - Inference settings · Controls top-k and score calibration behavior.
inference_settings.json - Dataset-ID prediction
predict.py - Raw-video prediction
predict_video.py - Evaluation and sweeps · Evaluates dataset splits and selects inference parameters.
evaluate.py
Sign-to-Sentence
Downstream keyword-to-text generation.
- Interactive sentence demo · Runs video-window recognition and sentence generation.
demo.py - Sign-to-sentence pipeline · Converts recognized signs to keywords and generates text candidates.
pipeline.py - T5 keyword generator · Separately trained keyword-to-sentence generator.
wts_split/inference.py
CTRGCN Experiment
Independent graph-recognition implementation.
- CTRGCN graph/model · Independent keypoint graph definition and CTRGCN model.
ctrgcn_workspace/model_ctrgcn.py - CTRGCN training
ctrgcn_workspace/train_ctrgcn.py - CTRGCN evaluation
ctrgcn_workspace/evaluate_ctrgcn.py
Connections 23
- Raw videos → Keypoint extraction: extracts
- Keypoint extraction → Sequence normalization: keypoint sequences
- Dataset and statistics → Dataset loader: samples and splits
- Dataset and statistics → Sequence normalization: global statistics
- Sequence normalization → Hierarchy training: normalized features
- Dataset loader → Hierarchy training: training samples
- Recognition model → Hierarchy training: architecture
- Hierarchy training → Hierarchy artifacts: writes mappings and checkpoints
- Hierarchy artifacts → Two-stage inference: loads hierarchy
- Inference settings → Two-stage inference: ranking controls
- Dataset loader → Dataset-ID prediction: dataset IDs
- Dataset-ID prediction → Two-stage inference: classifies
- Raw videos → Raw-video prediction: video input
- Raw-video prediction → Keypoint extraction: extracts features
- Sequence normalization → Two-stage inference: normalized video features
- Dataset loader → Evaluation and sweeps: evaluation splits
- Evaluation and sweeps → Two-stage inference: evaluates and sweeps
- Interactive sentence demo → Sign-to-sentence pipeline: runs
- Sign-to-sentence pipeline → Two-stage inference: recognized sign labels
- Sign-to-sentence pipeline → T5 keyword generator: keyword text
- Keypoint extraction → CTRGCN graph/model: keypoint representation
- CTRGCN graph/model → CTRGCN training: trains
- CTRGCN graph/model → CTRGCN evaluation: evaluates