How makeitezzz555/signai-asl-gesture-demo works
SignAI is a browser-based ASL gesture demo with a companion local dataset, training, and evaluation workflow. Its main runtime path is webcam landmarks → ONNX sequence recognition and gesture confirmation → displayed words and optional speech; face-touch labels are a separate recognition branch. The optional Hybrid mode adds browser-local TensorFlow.js predictions with ONNX fallback. Edges follow the README’s documented flow and the sampled module calls; page-level wiring not shown in excerpts is represented only where documented.
Main entry points: App.tsx App and Routes · RecognizePage.tsx Recognize Page · AppContext.tsx App State · modelEngine.ts TF.js Model Engine
Components 24
Application
- App and Routes
src/App.tsx
Live Recognition
- Recognize Page
src/pages/RecognizePage.tsx - Camera Tracking
src/hooks/useMediaPipe.ts - Landmark Features
src/utils/landmarks.ts - ONNX Inference
src/ml/inferenceModel.ts - Gesture Segmentation
src/ml/segmentationFSM.ts - Face Interactions
src/hooks/useMediaPipe.ts - Hybrid Router
src/ml/hybridRouter.ts - Prediction Smoothing
src/hooks/usePredictionSmoothing.ts
Custom Model Workflow
- Dataset Page
src/pages/DatasetPage.tsx - Train Page
src/pages/TrainPage.tsx - Evaluate Page
src/pages/EvaluatePage.tsx - Dataset Processing
src/dataset/datasetUtils.ts - TF.js Model Engine
src/ml/modelEngine.ts - Model State API
src/ml/model.ts - Model Metrics
src/ml/modelEngine.ts
Browser State
- App State
src/contexts/AppContext.tsx - Dataset Storage
src/dataset/datasetStorage.ts
Speech Output
- Speech Selection
src/utils/tts.ts - Local Speech
src/utils/espeakFallback.ts - Cloud TTS Client
src/utils/cloudTts.ts - TTS Proxy
server/tts/handler.ts
Other components
- User
- Google TTS
Connections 30
- User → App and Routes: uses
- App and Routes → Recognize Page: routes
- App and Routes → Dataset Page: routes
- App and Routes → Train Page: routes
- App and Routes → Evaluate Page: routes
- App and Routes → ONNX Inference: loads model
- Recognize Page → Camera Tracking: starts tracking
- Camera Tracking → Landmark Features: normalizes
- Camera Tracking → Face Interactions: detects proximity
- Landmark Features → ONNX Inference: provides features
- ONNX Inference → Gesture Segmentation: segments predictions
- Recognize Page → Hybrid Router: routes predictions
- Hybrid Router → ONNX Inference: uses fallback
- Recognize Page → Prediction Smoothing: confirms predictions
- Dataset Page → App State: edits dataset
- Train Page → App State: uses dataset
- Evaluate Page → App State: reads metrics
- App State → Dataset Storage: reads and writes
- App State → Model State API: subscribes to state
- Dataset Page → Dataset Processing: imports samples
- Train Page → Model State API: trains or loads
- Model State API → TF.js Model Engine: delegates model work
- TF.js Model Engine → Dataset Processing: encodes samples
- TF.js Model Engine → Model Metrics: computes metrics
- TF.js Model Engine → User: stores model locally
- Recognize Page → Speech Selection: requests speech
- Speech Selection → Local Speech: speaks locally
- Speech Selection → Cloud TTS Client: optionally requests
- Cloud TTS Client → TTS Proxy: sends text
- TTS Proxy → Google TTS: requests synthesis