abel1712/asl-recognition

How abel1712/asl-recognition works

This is an offline ASL word-to-speech translator. A signer’s webcam video is processed into landmark sequences, classified into words, stabilized over time, and accumulated until the user requests a sentence; grammar correction and speech synthesis then produce spoken English. A separate preparation and training workflow turns WLASL videos into augmented keypoints and trained models. The live app, smoother, grammar fixer, and TTS are evidenced in source excerpts; links across preparation and training stages follow the README workflow, and the documented video evaluator is unsampled.

Main entry points: app.py Live app · asl_model_final.pth Trained model files · temporal_smoother.py Temporal smoothing · tts_kokoro.py Speech synthesis

Components 21

Live recognition

Webcam video
Live app src/app.py
Landmark features src/app.py
Word classifier src/app.py
PyTorch weights models/asl_model_best.pth
Word index models/idx_to_word.json
Temporal smoothing src/temporal_smoother.py
Word buffer and controls src/app.py

Sentence and speech

Grammar correction src/grammar_fix.py
Speech synthesis src/tts_kokoro.py
Spoken sentence

Data preparation

WLASL videos
Video keypoint extraction src/extract_keypoints.py
Keypoint sequences
Keypoint augmentation src/augment_data.py
Augmented sequences

Model development

PyTorch training src/train_model_pytorch.py
Legacy Keras training src/train_model.py
Trained model files models/asl_model_final.pth
Video evaluation src/test_on_video.py

Other components

Signer

Connections 22

Signer → Webcam video: performs signs README.md
Webcam video → Live app: supplies frames src/app.py
Live app → Landmark features: extracts features src/app.py
Landmark features → Word classifier: provides sequences src/app.py
Word classifier → PyTorch weights: loads weights src/app.py
Word classifier → Word index: maps predictions src/app.py
Live app → Temporal smoothing: filters predictions src/app.py
Temporal smoothing → Word buffer and controls: commits words src/app.py
Live app → Word buffer and controls: handles controls src/app.py
Word buffer and controls → Grammar correction: formats sentence src/app.py
Word buffer and controls → Speech synthesis: requests speech src/app.py
Speech synthesis → Spoken sentence: speaks sentence README.md
WLASL videos → Video keypoint extraction: provides videos README.md
Video keypoint extraction → Keypoint sequences: saves sequences README.md
Keypoint sequences → Keypoint augmentation: provides samples README.md
Keypoint augmentation → Augmented sequences: writes variants README.md
Augmented sequences → PyTorch training: trains on samples README.md
PyTorch training → Trained model files: saves weights src/train_model_pytorch.py
Augmented sequences → Legacy Keras training: trains on samples README.md
Legacy Keras training → Trained model files: saves legacy weights src/train_model.py
Trained model files → Word classifier: supports inference README.md
Video evaluation → Trained model files: evaluates predictions README.md

More Python diagrams: joshuachou2018/autoba · kyaukyuai/gpt-all-star · maxapiteam/pymax · microsoft/tokeneconomizer · pengxiaoan/autocad-dwg-redraw-skill

Drawn by GitDiagram on September 30, 2026 from the repository's file tree, README and source files. Read as Markdown ·