How amulyakallakuri/real-time-alphabet-recognition-asl works
Project type and purpose
• This is a real-time computer-vision full-stack application for recognizing ASL alphabet letters from live video, evaluating multiple CNN backbones, and optionally autocorrecting and synthesizing speech output.
• It includes both a training pipeline (data preparation, model comparison, checkpointing) and an inference GUI pipeline (hand detection, classification, autocorrect, display/speech).
File structure
Top-level:
• data/ – raw and augmented image datasets for training and testing
• proposal.tex – project proposal document
• src/ – Python modules implementing data utilities, model definitions, training, inference, GUI, and spell-checker
• README.md – high-level project description
src/ modules of note:
• data_utils.py, split_train_val.py – data loading, augmentation, train/validation split
• cnn.py, ResNet50.py, VGG16.py, EfficientNetV2L.py – model definitions/wrappers
• trainer.py – orchestrates training loop, metrics logging, checkpoint saving
• test_inference.py – command-line inference on sample images
• handDetector.py – locates and crops hand region from camera frames
• GUI.py – real-time application entry point, connects video stream to inference and display
• Auto_Correct_SpellChecker.py, word_corrector.py – post-processing of recognized letters into words and autocorrect
Key insights from README
• Core task: static ASL alphabet recognition (29 classes + “non-ASL” class)
• Evaluation criteria: Top-1/Top-5 accuracy + inference FPS
• Multiple CNN backbones investigated; final model balances speed and accuracy
• Real-time feedback and optional text-to-speech output
Diagramming recommendations
• Interactions (e.g., “trainer.py writes models to /checkpoints,” “GUI.py calls handDetector.py”)
• Performance metrics near components (e.g., “Inference: 30 FPS on 3080Ti”)