How hurrypatter/asl-glasses works
ASL Smart Glasses performs on-device sign recognition, turning camera frames into committed text and optional speech. The live path detects one hand, recognizes J/Z from motion or static handshapes with a trained classifier, debounces commits, then corrects fingerspelled runs and expands word-sign tokens. Separate workflows collect signer-labeled landmarks, train the model, and evaluate the complete pipeline under recorded conditions. Some wiring is supported by README documentation rather than sampled source; unsampled details are omitted.
Main entry points: main.py Live application · evaluate.py End-to-end evaluation · collect_data.py Data collection · train_classifier.py Model training
Components 20
Live translation
- Live application
main.py - On-screen output
main.py
Recognition
- Hand Landmarker
- MediaPipe model
hand_landmarker.task - Motion detection
motion.py - Static classifier
landmark_model.joblib - Classifier labels
landmark_labels.json - Commit debouncer
debouncer.py
Text output
- Text bridge
nlp_bridge.py - Speech output
audio.py
Data and evaluation
- Data collection
collect_data.py - Dataset schema
dataset.py - Landmark data
landmark_data.csv - Model training
train_classifier.py - End-to-end evaluation
evaluate.py - Evaluation results
eval_results.csv
Other components
- Signer
- Camera
- SymSpell
- PowerShell speech
Connections 27
- Signer → Camera: signs to
- Camera → Live application: provides frames
- Live application → Hand Landmarker: detects hands
- Hand Landmarker → MediaPipe model: uses model
- Live application → Motion detection: checks trajectories
- Live application → Static classifier: classifies features
- Live application → Classifier labels: loads labels
- Live application → Commit debouncer: commits predictions
- Live application → Text bridge: corrects commits
- Text bridge → SymSpell: segments text
- Live application → Speech output: speaks text
- Speech output → PowerShell speech: invokes speech
- Live application → On-screen output: renders status
- Signer → Data collection: selects labels
- Data collection → Hand Landmarker: detects hands
- Data collection → Dataset schema: uses schema
- Data collection → Landmark data: appends features
- Model training → Landmark data: reads samples
- Model training → Static classifier: writes model
- Model training → Classifier labels: writes labels
- Signer → End-to-end evaluation: provides targets
- End-to-end evaluation → Hand Landmarker: detects hands
- End-to-end evaluation → Motion detection: checks trajectories
- End-to-end evaluation → Static classifier: classifies features
- End-to-end evaluation → Classifier labels: loads labels
- End-to-end evaluation → Commit debouncer: commits predictions
- End-to-end evaluation → Evaluation results: appends scores