# sahityabiswas/gesture2sentence architecture

> Architecture diagram of the GitHub repository sahityabiswas/gesture2sentence, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sahityabiswas/gesture2sentence
- Repository: https://github.com/sahityabiswas/gesture2sentence

## Overview

#### Purpose and Entry Points
Python-based sign-language recognition system for isolated signs, with optional sign-keyword-to-sentence generation. Primary operational entry points are training (`Train_hierarchical.py`), dataset-ID prediction (`predict.py`), raw-video prediction (`predict_video.py`), evaluation (`evaluate.py`), and interactive sentence demo (`demo.py`).

#### Keypoint Input and Feature Boundary
The core classifier consumes normalized keypoint sequences rather than video pixels. Raw videos pass through MediaPipe-based extraction and sequence preprocessing before normalization against persisted global statistics. Dataset-backed samples are loaded directly from stored keypoint data.
Grounding: `video_keypoint_extractor.py`, `video_preprocess_utils.py`, `normalize.py`, `dataset.py`, `global_stats.pkl`.

#### Hierarchical Recognition Training
`Train_hierarchical.py` orchestrates creation of label mappings, class-to-group clustering, group-model training, and one class submodel per group. The hierarchy reduces a large flat sign label space into group routing followed by within-group classification. Model structure and training dimensions are defined by the main model/config layer.
Grounding: `Train_hierarchical.py`, `model.py`, `config.py`, `label_map.pkl`, `group_map.pkl`.

#### Two-Stage Inference
Inference first scores sign groups, then evaluates class candidates within selected groups, combining/ranking predictions using configurable settings such as top-k and score calibration. Saved group/submodel checkpoints and mapping artifacts are an explicit boundary between training and serving; artifact location can be redirected through environment configuration.
Grounding: `hierarchical_inference.py`, `inference_utils.py`, `inference_settings.json`, `predict.py`, `predict_video.py`.

#### Evaluation and Artifact Lifecycle
Evaluation runs the trained hierarchy against dataset splits and supports parameter sweeps to select inference settings. Persisted data mappings, normalization statistics, group assignments, and omitted PyTorch checkpoints are required runtime dependencies; source control includes only small mapping/statistical artifacts visible in the tree.
Grounding: `evaluate.py`, `dataset.csv`, `vid_class_FDMSE-ISL.pkl`, `vid_splits_FDMSE-ISL.pkl`, `class_map_FDMSE-ISL.csv`.

#### Sign-to-Sentence Generation
A separate downstream pipeline predicts signs over a short video window, converts labels to keyword text, and invokes a T5 text generator to generate natural-language candidates. T5 training and direct keyword inference are isolated under `wts_split`; this stage depends on the recognition output but is not part of classifier training.
Grounding: `pipeline.py`, `demo.py`, `wts_split/training.py`, `wts_split/inference.py`, `wts_split/dataset_word_gen.py`.

#### Alternate Graph-Model Experiment
`ctrgcn_workspace` is a separate CTRGCN-based graph-recognition experiment that shares the keypoint representation concept but has independent dataset, graph definition, model, training, inference, and evaluation modules. It should be treated as an experimental parallel architecture, not as a dependency of the main hierarchical serving path.
Grounding: `ctrgcn_workspace/graph.py`, `ctrgcn_workspace/model_ctrgcn.py`, `ctrgcn_workspace/train_ctrgcn.py`, `ctrgcn_workspace/evaluate_ctrgcn.py`.

#### Runtime and External Dependencies
The runtime is Python with PyTorch for recognition models, NumPy/Pandas/scikit-learn for data and KMeans grouping, OpenCV and MediaPipe for video/keypoint extraction, and Hugging Face Transformers/T5 for language generation. CUDA-capable PyTorch is optional for acceleration; no web service, database, or deployment infrastructure is represented.

Main entry points: [`hierarchical_inference.py`](https://github.com/sahityabiswas/gesture2sentence/blob/main/hierarchical_inference.py) (Two-stage inference), [`dataset.py`](https://github.com/sahityabiswas/gesture2sentence/blob/main/dataset.py) (Dataset loader), [`video_keypoint_extractor.py`](https://github.com/sahityabiswas/gesture2sentence/blob/main/video_keypoint_extractor.py) (Keypoint extraction), [`normalize.py`](https://github.com/sahityabiswas/gesture2sentence/blob/main/normalize.py) (Sequence normalization)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Keypoint Data"]
  node_raw_video["Raw videos<br/>video input"]
  node_keypoint_extraction["Keypoint extraction<br/>MediaPipe preprocessing"]
  node_normalization["Sequence normalization<br/>feature transform<br/>[normalize.py]"]
  node_dataset_loader["Dataset loader<br/>PyTorch dataset<br/>[dataset.py]"]
  node_dataset_artifacts[("Dataset and statistics<br/>persisted data<br/>[dataset.csv]")]
end

subgraph group_recognition["Hierarchical Recognition"]
  node_training["Hierarchy training<br/>training entry point"]
  node_model_config["Recognition model<br/>PyTorch model/config<br/>[model.py]"]
  node_hierarchy_artifacts[("Hierarchy artifacts<br/>model artifacts<br/>[label_map.pkl]")]
  node_inference_engine["Two-stage inference<br/>hierarchical scorer"]
  node_inference_settings["Inference settings<br/>serving configuration"]
  node_dataset_prediction["Dataset-ID prediction<br/>CLI entry point<br/>[predict.py]"]
  node_video_prediction["Raw-video prediction<br/>CLI entry point<br/>[predict_video.py]"]
  node_evaluation["Evaluation and sweeps<br/>evaluation entry point<br/>[evaluate.py]"]
end

subgraph group_language["Sign-to-Sentence"]
  node_sentence_demo["Interactive sentence demo<br/>demo entry point<br/>[demo.py]"]
  node_sign_sentence_pipeline["Sign-to-sentence pipeline<br/>[pipeline.py]"]
  node_t5_generation["T5 keyword generator<br/>Transformers model<br/>[inference.py]"]
end

subgraph group_experiment["CTRGCN Experiment"]
  node_ctrgcn_graph["CTRGCN graph/model<br/>graph recognition model<br/>[model_ctrgcn.py]"]
  node_ctrgcn_training["CTRGCN training<br/>experiment training<br/>[train_ctrgcn.py]"]
  node_ctrgcn_evaluation["CTRGCN evaluation<br/>experiment evaluation<br/>[evaluate_ctrgcn.py]"]
end

node_raw_video -->|"extracts"| node_keypoint_extraction
node_keypoint_extraction -->|"keypoint sequences"| node_normalization
node_dataset_artifacts -->|"samples and splits"| node_dataset_loader
node_dataset_artifacts -->|"global statistics"| node_normalization
node_normalization -->|"normalized features"| node_training
node_dataset_loader -->|"training samples"| node_training
node_model_config -->|"architecture"| node_training
node_training -->|"writes mappings and checkpoints"| node_hierarchy_artifacts
node_hierarchy_artifacts -->|"loads hierarchy"| node_inference_engine
node_inference_settings -->|"ranking controls"| node_inference_engine
node_dataset_loader -->|"dataset IDs"| node_dataset_prediction
node_dataset_prediction -->|"classifies"| node_inference_engine
node_raw_video -->|"video input"| node_video_prediction
node_video_prediction -->|"extracts features"| node_keypoint_extraction
node_normalization -->|"normalized video features"| node_inference_engine
node_dataset_loader -->|"evaluation splits"| node_evaluation
node_evaluation -->|"evaluates and sweeps"| node_inference_engine
node_sentence_demo -->|"runs"| node_sign_sentence_pipeline
node_sign_sentence_pipeline -->|"recognized sign labels"| node_inference_engine
node_sign_sentence_pipeline -->|"keyword text"| node_t5_generation
node_keypoint_extraction -.->|"keypoint representation"| node_ctrgcn_graph
node_ctrgcn_graph -->|"trains"| node_ctrgcn_training
node_ctrgcn_graph -->|"evaluates"| node_ctrgcn_evaluation

click node_keypoint_extraction "https://github.com/sahityabiswas/gesture2sentence/blob/main/video_keypoint_extractor.py"
click node_normalization "https://github.com/sahityabiswas/gesture2sentence/blob/main/normalize.py"
click node_dataset_loader "https://github.com/sahityabiswas/gesture2sentence/blob/main/dataset.py"
click node_dataset_artifacts "https://github.com/sahityabiswas/gesture2sentence/blob/main/dataset.csv"
click node_training "https://github.com/sahityabiswas/gesture2sentence/blob/main/Train_hierarchical.py"
click node_model_config "https://github.com/sahityabiswas/gesture2sentence/blob/main/model.py"
click node_hierarchy_artifacts "https://github.com/sahityabiswas/gesture2sentence/blob/main/label_map.pkl"
click node_inference_engine "https://github.com/sahityabiswas/gesture2sentence/blob/main/hierarchical_inference.py"
click node_inference_settings "https://github.com/sahityabiswas/gesture2sentence/blob/main/inference_settings.json"
click node_dataset_prediction "https://github.com/sahityabiswas/gesture2sentence/blob/main/predict.py"
click node_video_prediction "https://github.com/sahityabiswas/gesture2sentence/blob/main/predict_video.py"
click node_evaluation "https://github.com/sahityabiswas/gesture2sentence/blob/main/evaluate.py"
click node_sentence_demo "https://github.com/sahityabiswas/gesture2sentence/blob/main/demo.py"
click node_sign_sentence_pipeline "https://github.com/sahityabiswas/gesture2sentence/blob/main/pipeline.py"
click node_t5_generation "https://github.com/sahityabiswas/gesture2sentence/blob/main/wts_split/inference.py"
click node_ctrgcn_graph "https://github.com/sahityabiswas/gesture2sentence/blob/main/ctrgcn_workspace/model_ctrgcn.py"
click node_ctrgcn_training "https://github.com/sahityabiswas/gesture2sentence/blob/main/ctrgcn_workspace/train_ctrgcn.py"
click node_ctrgcn_evaluation "https://github.com/sahityabiswas/gesture2sentence/blob/main/ctrgcn_workspace/evaluate_ctrgcn.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw_video,node_keypoint_extraction,node_normalization,node_dataset_loader,node_dataset_artifacts toneBlue
class node_training,node_model_config,node_hierarchy_artifacts,node_inference_engine,node_inference_settings,node_dataset_prediction,node_video_prediction,node_evaluation toneAmber
class node_sentence_demo,node_sign_sentence_pipeline,node_t5_generation toneMint
class node_ctrgcn_graph,node_ctrgcn_training,node_ctrgcn_evaluation toneRose
```

## Components

### Keypoint Data

Shared input representation and persisted dataset artifacts.

- **Raw videos**: video input
- [Keypoint extraction](https://github.com/sahityabiswas/gesture2sentence/blob/main/video_keypoint_extractor.py): MediaPipe preprocessing · Extracts and preprocesses video pose/keypoint sequences. · `video_keypoint_extractor.py`
- [Sequence normalization](https://github.com/sahityabiswas/gesture2sentence/blob/main/normalize.py): feature transform · Normalizes keypoint sequences with persisted global statistics. · `normalize.py`
- [Dataset loader](https://github.com/sahityabiswas/gesture2sentence/blob/main/dataset.py): PyTorch dataset · Loads stored keypoint samples and dataset splits. · `dataset.py`
- [Dataset and statistics](https://github.com/sahityabiswas/gesture2sentence/blob/main/dataset.csv): persisted data · Dataset metadata, splits, class map, keypoints, and global statistics. · `dataset.csv`

### Hierarchical Recognition

Primary sign classifier training, artifacts, and serving.

- [Hierarchy training](https://github.com/sahityabiswas/gesture2sentence/blob/main/Train_hierarchical.py): training entry point · Builds label/group mappings and trains routing and group-specific classifiers. · `Train_hierarchical.py`
- [Recognition model](https://github.com/sahityabiswas/gesture2sentence/blob/main/model.py): PyTorch model/config · Defines classifier structure and training dimensions. · `model.py`
- [Hierarchy artifacts](https://github.com/sahityabiswas/gesture2sentence/blob/main/label_map.pkl): model artifacts · Label/group mappings and saved group/submodel checkpoints. · `label_map.pkl`
- [Two-stage inference](https://github.com/sahityabiswas/gesture2sentence/blob/main/hierarchical_inference.py): hierarchical scorer · Routes through sign groups then ranks within-group class candidates. · `hierarchical_inference.py`
- [Inference settings](https://github.com/sahityabiswas/gesture2sentence/blob/main/inference_settings.json): serving configuration · Controls top-k and score calibration behavior. · `inference_settings.json`
- [Dataset-ID prediction](https://github.com/sahityabiswas/gesture2sentence/blob/main/predict.py): CLI entry point · `predict.py`
- [Raw-video prediction](https://github.com/sahityabiswas/gesture2sentence/blob/main/predict_video.py): CLI entry point · `predict_video.py`
- [Evaluation and sweeps](https://github.com/sahityabiswas/gesture2sentence/blob/main/evaluate.py): evaluation entry point · Evaluates dataset splits and selects inference parameters. · `evaluate.py`

### Sign-to-Sentence

Downstream keyword-to-text generation.

- [Interactive sentence demo](https://github.com/sahityabiswas/gesture2sentence/blob/main/demo.py): demo entry point · Runs video-window recognition and sentence generation. · `demo.py`
- [Sign-to-sentence pipeline](https://github.com/sahityabiswas/gesture2sentence/blob/main/pipeline.py): pipeline · Converts recognized signs to keywords and generates text candidates. · `pipeline.py`
- [T5 keyword generator](https://github.com/sahityabiswas/gesture2sentence/blob/main/wts_split/inference.py): Transformers model · Separately trained keyword-to-sentence generator. · `wts_split/inference.py`

### CTRGCN Experiment

Independent graph-recognition implementation.

- [CTRGCN graph/model](https://github.com/sahityabiswas/gesture2sentence/blob/main/ctrgcn_workspace/model_ctrgcn.py): graph recognition model · Independent keypoint graph definition and CTRGCN model. · `ctrgcn_workspace/model_ctrgcn.py`
- [CTRGCN training](https://github.com/sahityabiswas/gesture2sentence/blob/main/ctrgcn_workspace/train_ctrgcn.py): experiment training · `ctrgcn_workspace/train_ctrgcn.py`
- [CTRGCN evaluation](https://github.com/sahityabiswas/gesture2sentence/blob/main/ctrgcn_workspace/evaluate_ctrgcn.py): experiment evaluation · `ctrgcn_workspace/evaluate_ctrgcn.py`

## Connections

- Raw videos → Keypoint extraction: extracts
- Keypoint extraction → Sequence normalization: keypoint sequences
- Dataset and statistics → Dataset loader: samples and splits
- Dataset and statistics → Sequence normalization: global statistics
- Sequence normalization → Hierarchy training: normalized features
- Dataset loader → Hierarchy training: training samples
- Recognition model → Hierarchy training: architecture
- Hierarchy training → Hierarchy artifacts: writes mappings and checkpoints
- Hierarchy artifacts → Two-stage inference: loads hierarchy
- Inference settings → Two-stage inference: ranking controls
- Dataset loader → Dataset-ID prediction: dataset IDs
- Dataset-ID prediction → Two-stage inference: classifies
- Raw videos → Raw-video prediction: video input
- Raw-video prediction → Keypoint extraction: extracts features
- Sequence normalization → Two-stage inference: normalized video features
- Dataset loader → Evaluation and sweeps: evaluation splits
- Evaluation and sweeps → Two-stage inference: evaluates and sweeps
- Interactive sentence demo → Sign-to-sentence pipeline: runs
- Sign-to-sentence pipeline → Two-stage inference: recognized sign labels
- Sign-to-sentence pipeline → T5 keyword generator: keyword text
- Keypoint extraction → CTRGCN graph/model: keypoint representation
- CTRGCN graph/model → CTRGCN training: trains
- CTRGCN graph/model → CTRGCN evaluation: evaluates