How daniyaa1/sign2sentence works
AVA provides two webcam inference modes: a single-sign recognizer and a continuous sentence translator. Both capture video, detect hand landmarks, prepare sequences, and classify them with the LSTM; the continuous mode stabilizes recognized glosses and turns them into English using a fast dictionary or Gemini. The README documents these capabilities, while the excerpts show key runtime calls. The continuous-mode sentence dispatch and some asset-loading details are only partially excerpted, so those links rely on documented behavior where noted.
Main entry points: predict.py Single-sign mode · predict_continuous.py Continuous mode · best_model.h5 LSTM classifier · label_map.npy Gesture labels
Components 17
Capture and features
- Single-sign mode
predict.py - Continuous mode
predict_continuous.py - Hand tracking
predict.py - Sequence preprocessing
predict.py - Sequence preprocessing
predict_continuous.py
Gesture recognition
- LSTM classifier
model/best_model.h5 - Gloss stabilization
predict_continuous.py - Prediction display
predict.py
Sentence translation
- Fast phrase dictionary
predict_continuous.py - Gemini grammar engine
predict_continuous.py - Sentence display
predict_continuous.py
Inference assets
- Gesture labels
output/label_map.npy - Feature scaler
output/scaler.pkl - Landmarker model
hand_landmarker.task
Other components
- Signer
- Webcam
- Google Gemini
Connections 23
- Signer → Webcam: signs to
README.md - Webcam → Single-sign mode: provides frames
predict.py - Webcam → Continuous mode: provides frames
predict_continuous.py - Single-sign mode → Hand tracking: detects hands
predict.py - Continuous mode → Hand tracking: detects hands
predict_continuous.py - Landmarker model → Hand tracking: configures
predict.py - Hand tracking → Sequence preprocessing: extracts keypoints
predict.py - Hand tracking → Sequence preprocessing: extracts keypoints
predict_continuous.py - Single-sign mode → Sequence preprocessing: preprocesses sequence
predict.py - Continuous mode → Sequence preprocessing: preprocesses sequence
predict_continuous.py - Sequence preprocessing → Feature scaler: scales with
predict.py - Sequence preprocessing → Feature scaler: scales with
predict_continuous.py - Single-sign mode → LSTM classifier: predicts sequence
predict.py - Continuous mode → LSTM classifier: predicts sequence
predict_continuous.py - Single-sign mode → Gesture labels: maps classes
predict.py - Continuous mode → Gesture labels: maps classes
README.md - Single-sign mode → Prediction display: shows prediction
predict.py - Continuous mode → Gloss stabilization: stabilizes words
predict_continuous.py - Gloss stabilization → Fast phrase dictionary: looks up phrase
predict_continuous.py - Gloss stabilization → Gemini grammar engine: translates unknown
predict_continuous.py - Gemini grammar engine → Google Gemini: requests sentence
predict_continuous.py - Fast phrase dictionary → Sentence display: supplies sentence
README.md - Gemini grammar engine → Sentence display: supplies sentence
README.md