How ishabrato/visual-voice works
Visual Voice recognizes ASL fingerspelling from a webcam and turns stable, confident letter/SPACE predictions into editable sentence text. The primary user path is the PyQt UI: capture frames, detect hand landmarks, normalize them, classify gestures, stabilize entries, then display and edit the sentence. Separate OpenCV scripts provide gesture-data logging and classifier demos; the README also describes a notebook-based training workflow. The UI and scripts are excerpted, but some internal setup is omitted, so only documented or directly visible links are drawn.
Main entry points: ui.py Live recognition UI · keypoint_classifier.py Keypoint classifier · app.py Gesture data logging · point_history_classifier.py Point-history classifier
Components 18
Capture and perception
- Live recognition UI
ui.py - Hand tracking and landmarks
ui.py - Landmark normalization
ui.py - OpenCV gesture demo
app.py - Alternate gesture demo
app2.py
Recognition models
- Keypoint classifier
model/keypoint_classifier/keypoint_classifier.py - Keypoint TFLite model
model/keypoint_classifier/keypoint_classifier.tflite - Point-history classifier
model/point_history_classifier/point_history_classifier.py - Point-history TFLite model
model/point_history_classifier/point_history_classifier.tflite
Sentence interface
Data and training
- Gesture data logging
app.py - Keypoint training data
model/keypoint_classifier/keypoint.csv - Point-history training data
model/point_history_classifier/point_history.csv - Model training notebook
keypoint_classification_EN.ipynb
Other components
- ASL signer
- Webcam
Connections 17
- ASL signer → Webcam: shows gestures
README.md - Webcam → Live recognition UI: provides frames
ui.py - Live recognition UI → Hand tracking and landmarks: processes frames
ui.py - Hand tracking and landmarks → Landmark normalization: extracts coordinates
README.md - Landmark normalization → Keypoint classifier: feeds landmarks
README.md - Live recognition UI → Keypoint classifier: classifies landmarks
ui.py - Keypoint classifier → Keypoint TFLite model: invokes
model/keypoint_classifier/keypoint_classifier.py - Keypoint classifier → Stability and sentence builder: returns gesture and confidence
ui.py - Stability and sentence builder → Video and sentence display: updates text
ui.py - Copy, clear, backspace → Stability and sentence builder: edits sentence
ui.py - Live recognition UI → Video and sentence display: renders video
ui.py - OpenCV gesture demo → Point-history classifier: classifies point history
app.py - Point-history classifier → Point-history TFLite model: invokes
model/point_history_classifier/point_history_classifier.py - Gesture data logging → Keypoint training data: appends keypoints
app.py - Gesture data logging → Point-history training data: appends history
app.py - Keypoint training data → Model training notebook: training input
README.md - Model training notebook → Keypoint TFLite model: exports inference model
README.md