# abel1712/asl-recognition architecture

> Architecture diagram of the GitHub repository abel1712/asl-recognition, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abel1712/asl-recognition
- Repository: https://github.com/abel1712/asl-recognition

## Overview

This is an offline ASL word-to-speech translator. A signer’s webcam video is processed into landmark sequences, classified into words, stabilized over time, and accumulated until the user requests a sentence; grammar correction and speech synthesis then produce spoken English. A separate preparation and training workflow turns WLASL videos into augmented keypoints and trained models. The live app, smoother, grammar fixer, and TTS are evidenced in source excerpts; links across preparation and training stages follow the README workflow, and the documented video evaluator is unsampled.

Main entry points: [`src/app.py`](https://github.com/abel1712/asl-recognition/blob/main/src/app.py) (Live app), [`models/asl_model_final.pth`](https://github.com/abel1712/asl-recognition/blob/main/models/asl_model_final.pth) (Trained model files), [`src/temporal_smoother.py`](https://github.com/abel1712/asl-recognition/blob/main/src/temporal_smoother.py) (Temporal smoothing), [`src/tts_kokoro.py`](https://github.com/abel1712/asl-recognition/blob/main/src/tts_kokoro.py) (Speech synthesis)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_live["Live recognition"]
  node_webcam["Webcam video"]
  node_live_app["Live app<br/>[app.py]"]
  node_landmarks["Landmark features<br/>[app.py]"]
  node_classifier["Word classifier<br/>[app.py]"]
  node_weights["PyTorch weights<br/>[asl_model_best.pth]"]
  node_label_index["Word index<br/>[idx_to_word.json]"]
  node_smoother["Temporal smoothing"]
  node_word_buffer["Word buffer and controls<br/>[app.py]"]
end

subgraph group_language["Sentence and speech"]
  node_grammar["Grammar correction<br/>[grammar_fix.py]"]
  node_speech["Speech synthesis<br/>[tts_kokoro.py]"]
  node_listener(("Spoken sentence"))
end

subgraph group_preparation["Data preparation"]
  node_wlasl["WLASL videos"]
  node_keypoint_extraction["Video keypoint extraction"]
  node_raw_keypoints["Keypoint sequences"]
  node_augmentation["Keypoint augmentation<br/>[augment_data.py]"]
  node_augmented_keypoints["Augmented sequences"]
end

subgraph group_training["Model development"]
  node_pytorch_training["PyTorch training"]
  node_keras_training["Legacy Keras training<br/>[train_model.py]"]
  node_trained_models["Trained model files"]
  node_video_evaluation["Video evaluation<br/>[test_on_video.py]"]
end

node_signer(("Signer"))

node_signer -->|"performs signs"| node_webcam
node_webcam -->|"supplies frames"| node_live_app
node_live_app -->|"extracts features"| node_landmarks
node_landmarks -->|"provides sequences"| node_classifier
node_classifier -->|"loads weights"| node_weights
node_classifier -->|"maps predictions"| node_label_index
node_live_app -->|"filters predictions"| node_smoother
node_smoother -->|"commits words"| node_word_buffer
node_live_app -->|"handles controls"| node_word_buffer
node_word_buffer -->|"formats sentence"| node_grammar
node_word_buffer -->|"requests speech"| node_speech
node_speech -->|"speaks sentence"| node_listener
node_wlasl -->|"provides videos"| node_keypoint_extraction
node_keypoint_extraction -->|"saves sequences"| node_raw_keypoints
node_raw_keypoints -->|"provides samples"| node_augmentation
node_augmentation -->|"writes variants"| node_augmented_keypoints
node_augmented_keypoints -->|"trains on samples"| node_pytorch_training
node_pytorch_training -->|"saves weights"| node_trained_models
node_augmented_keypoints -->|"trains on samples"| node_keras_training
node_keras_training -->|"saves legacy weights"| node_trained_models
node_trained_models -.->|"supports inference"| node_classifier
node_video_evaluation -.->|"evaluates predictions"| node_trained_models

click node_live_app "https://github.com/abel1712/asl-recognition/blob/main/src/app.py"
click node_landmarks "https://github.com/abel1712/asl-recognition/blob/main/src/app.py"
click node_classifier "https://github.com/abel1712/asl-recognition/blob/main/src/app.py"
click node_weights "https://github.com/abel1712/asl-recognition/blob/main/models/asl_model_best.pth"
click node_label_index "https://github.com/abel1712/asl-recognition/blob/main/models/idx_to_word.json"
click node_smoother "https://github.com/abel1712/asl-recognition/blob/main/src/temporal_smoother.py"
click node_word_buffer "https://github.com/abel1712/asl-recognition/blob/main/src/app.py"
click node_grammar "https://github.com/abel1712/asl-recognition/blob/main/src/grammar_fix.py"
click node_speech "https://github.com/abel1712/asl-recognition/blob/main/src/tts_kokoro.py"
click node_keypoint_extraction "https://github.com/abel1712/asl-recognition/blob/main/src/extract_keypoints.py"
click node_augmentation "https://github.com/abel1712/asl-recognition/blob/main/src/augment_data.py"
click node_pytorch_training "https://github.com/abel1712/asl-recognition/blob/main/src/train_model_pytorch.py"
click node_keras_training "https://github.com/abel1712/asl-recognition/blob/main/src/train_model.py"
click node_trained_models "https://github.com/abel1712/asl-recognition/blob/main/models/asl_model_final.pth"
click node_video_evaluation "https://github.com/abel1712/asl-recognition/blob/main/src/test_on_video.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_webcam,node_live_app,node_landmarks,node_classifier,node_weights,node_label_index,node_smoother,node_word_buffer toneBlue
class node_grammar,node_speech,node_listener toneAmber
class node_wlasl,node_keypoint_extraction,node_raw_keypoints,node_augmentation,node_augmented_keypoints toneMint
class node_pytorch_training,node_keras_training,node_trained_models,node_video_evaluation toneRose
class node_signer toneIndigo
```

## Components

### Live recognition

- **Webcam video**
- [Live app](https://github.com/abel1712/asl-recognition/blob/main/src/app.py): `src/app.py`
- [Landmark features](https://github.com/abel1712/asl-recognition/blob/main/src/app.py): `src/app.py`
- [Word classifier](https://github.com/abel1712/asl-recognition/blob/main/src/app.py): `src/app.py`
- [PyTorch weights](https://github.com/abel1712/asl-recognition/blob/main/models/asl_model_best.pth): `models/asl_model_best.pth`
- [Word index](https://github.com/abel1712/asl-recognition/blob/main/models/idx_to_word.json): `models/idx_to_word.json`
- [Temporal smoothing](https://github.com/abel1712/asl-recognition/blob/main/src/temporal_smoother.py): `src/temporal_smoother.py`
- [Word buffer and controls](https://github.com/abel1712/asl-recognition/blob/main/src/app.py): `src/app.py`

### Sentence and speech

- [Grammar correction](https://github.com/abel1712/asl-recognition/blob/main/src/grammar_fix.py): `src/grammar_fix.py`
- [Speech synthesis](https://github.com/abel1712/asl-recognition/blob/main/src/tts_kokoro.py): `src/tts_kokoro.py`
- **Spoken sentence**

### Data preparation

- **WLASL videos**
- [Video keypoint extraction](https://github.com/abel1712/asl-recognition/blob/main/src/extract_keypoints.py): `src/extract_keypoints.py`
- **Keypoint sequences**
- [Keypoint augmentation](https://github.com/abel1712/asl-recognition/blob/main/src/augment_data.py): `src/augment_data.py`
- **Augmented sequences**

### Model development

- [PyTorch training](https://github.com/abel1712/asl-recognition/blob/main/src/train_model_pytorch.py): `src/train_model_pytorch.py`
- [Legacy Keras training](https://github.com/abel1712/asl-recognition/blob/main/src/train_model.py): `src/train_model.py`
- [Trained model files](https://github.com/abel1712/asl-recognition/blob/main/models/asl_model_final.pth): `models/asl_model_final.pth`
- [Video evaluation](https://github.com/abel1712/asl-recognition/blob/main/src/test_on_video.py): `src/test_on_video.py`

### Other components

- **Signer**

## Connections

- Signer → Webcam video: performs signs (evidence: `README.md`)
- Webcam video → Live app: supplies frames (evidence: `src/app.py`)
- Live app → Landmark features: extracts features (evidence: `src/app.py`)
- Landmark features → Word classifier: provides sequences (evidence: `src/app.py`)
- Word classifier → PyTorch weights: loads weights (evidence: `src/app.py`)
- Word classifier → Word index: maps predictions (evidence: `src/app.py`)
- Live app → Temporal smoothing: filters predictions (evidence: `src/app.py`)
- Temporal smoothing → Word buffer and controls: commits words (evidence: `src/app.py`)
- Live app → Word buffer and controls: handles controls (evidence: `src/app.py`)
- Word buffer and controls → Grammar correction: formats sentence (evidence: `src/app.py`)
- Word buffer and controls → Speech synthesis: requests speech (evidence: `src/app.py`)
- Speech synthesis → Spoken sentence: speaks sentence (evidence: `README.md`)
- WLASL videos → Video keypoint extraction: provides videos (evidence: `README.md`)
- Video keypoint extraction → Keypoint sequences: saves sequences (evidence: `README.md`)
- Keypoint sequences → Keypoint augmentation: provides samples (evidence: `README.md`)
- Keypoint augmentation → Augmented sequences: writes variants (evidence: `README.md`)
- Augmented sequences → PyTorch training: trains on samples (evidence: `README.md`)
- PyTorch training → Trained model files: saves weights (evidence: `src/train_model_pytorch.py`)
- Augmented sequences → Legacy Keras training: trains on samples (evidence: `README.md`)
- Legacy Keras training → Trained model files: saves legacy weights (evidence: `src/train_model.py`)
- Trained model files → Word classifier: supports inference (evidence: `README.md`)
- Video evaluation → Trained model files: evaluates predictions (evidence: `README.md`)