How buriburi-zaeemon/unmute works
Unmute is a bilingual ASL/ISL sign-language translation application. Its central workflow sends live camera frames or uploaded video from a browser to a FastAPI service, which extracts hand landmarks, recognizes signs, and returns translation results; the documented pipeline also includes temporal filtering and sentence processing. The repository includes separate custom-gesture, practice/dictionary/3D-guide, and static-model training capabilities. Some runtime wiring and the frontend’s internal feature boundaries are not shown in the sampled source, so those relationships are omitted or marked unverified.
Main entry points: main.py FastAPI service · video_processor.py Video translation · sentence_processor.py Sentence processor · app.js Web application
Components 22
Browser experience
- Web application
static/app.js - Practice and dictionary
static/app.js - 3D sign guide
static/app.js
Translation service
- FastAPI service
backend/main.py - Video translation
sign_engine/video_processor.py
Sign recognition
- Hand landmarks
sign_engine/landmark_extractor.py - Feature engineering
sign_engine/feature_engineering.py - ASL classifier
sign_engine/asl_classifier.py - Temporal gestures
sign_engine/temporal_tracker.py - Custom gesture trainer
sign_engine/gesture_trainer.py
Sentence processing
- Token contract
nlp/token_schema.py - Token normalization
nlp/token_normalizer.py - Sequence buffer
nlp/sequence_buffer.py - Sentence processor
nlp/sentence_processor.py - Grammar rules
nlp/grammar_rules.py - Language vocabulary
nlp/vocabulary.py
Model development
- Static sign datasets
ml/data/dataset.py - Class labels
ml/data/labels.py - Static model training
ml/training/train_static.py - Static MLP models
ml/models/static_mlp.py
Other components
- Signer
- Video file
Connections 33
- Signer → Web application: uses
README.md - Web application → FastAPI service: sends frames
README.md - FastAPI service → Web application: returns results
README.md - Signer → Video file: provides
README.md - Web application → FastAPI service: uploads video
README.md - FastAPI service → Video translation: processes video
backend/main.py - Video file → Video translation: is analyzed by
sign_engine/video_processor.py - FastAPI service → Hand landmarks: invokes
backend/main.py - FastAPI service → Feature engineering: invokes
backend/main.py - FastAPI service → ASL classifier: invokes
backend/main.py - FastAPI service → Temporal gestures: invokes
backend/main.py - FastAPI service → Custom gesture trainer: invokes
backend/main.py - Video translation → Hand landmarks: extracts
sign_engine/video_processor.py - Video translation → Feature engineering: computes
sign_engine/video_processor.py - Video translation → ASL classifier: classifies
sign_engine/video_processor.py - Video translation → Temporal gestures: tracks gestures
sign_engine/video_processor.py - Video translation → Custom gesture trainer: uses trainer
sign_engine/video_processor.py - ASL classifier → Feature engineering: extracts features
sign_engine/asl_classifier.py - ASL classifier → Hand landmarks: accepts landmarks
sign_engine/asl_classifier.py - FastAPI service → Sentence processor: processes recognized text
- Sentence processor → Token normalization: normalizes with
nlp/sentence_processor.py - Sentence processor → Sequence buffer: buffers with
nlp/sentence_processor.py - Sentence processor → Grammar rules: applies rules
nlp/sentence_processor.py - Sentence processor → Language vocabulary: uses vocabulary
nlp/sentence_processor.py - Token normalization → Token contract: normalizes
nlp/token_normalizer.py - Sequence buffer → Token contract: stores
nlp/sequence_buffer.py - Grammar rules → Token contract: transforms
nlp/grammar_rules.py - Grammar rules → Language vocabulary: uses vocabulary
nlp/grammar_rules.py - Static model training → Static sign datasets: loads batches
ml/training/train_static.py - Static model training → Static MLP models: trains
ml/training/train_static.py - Static sign datasets → Class labels: maps classes
ml/data/dataset.py - Static model training → Class labels: selects classes
ml/training/train_static.py - Practice and dictionary → 3D sign guide: offers sign guides
README.md