# makeitezzz555/signai-asl-gesture-demo architecture

> Architecture diagram of the GitHub repository makeitezzz555/signai-asl-gesture-demo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/makeitezzz555/signai-asl-gesture-demo
- Repository: https://github.com/makeitezzz555/signai-asl-gesture-demo

## Overview

SignAI is a browser-based ASL gesture demo with a companion local dataset, training, and evaluation workflow. Its main runtime path is webcam landmarks → ONNX sequence recognition and gesture confirmation → displayed words and optional speech; face-touch labels are a separate recognition branch. The optional Hybrid mode adds browser-local TensorFlow.js predictions with ONNX fallback. Edges follow the README’s documented flow and the sampled module calls; page-level wiring not shown in excerpts is represented only where documented.

Main entry points: [`src/App.tsx`](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/App.tsx) (App and Routes), [`src/pages/RecognizePage.tsx`](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/pages/RecognizePage.tsx) (Recognize Page), [`src/contexts/AppContext.tsx`](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/contexts/AppContext.tsx) (App State), [`src/ml/modelEngine.ts`](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/ml/modelEngine.ts) (TF.js Model Engine)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_app["Application"]
  node_app["App and Routes<br/>[App.tsx]"]
end

subgraph group_recognition["Live Recognition"]
  node_recognize["Recognize Page<br/>[RecognizePage.tsx]"]
  node_mediapipe["Camera Tracking<br/>[useMediaPipe.ts]"]
  node_landmarks["Landmark Features<br/>[landmarks.ts]"]
  node_onnx["ONNX Inference<br/>[inferenceModel.ts]"]
  node_fsm["Gesture Segmentation<br/>[segmentationFSM.ts]"]
  node_face["Face Interactions<br/>[useMediaPipe.ts]"]
  node_hybrid["Hybrid Router<br/>[hybridRouter.ts]"]
  node_smoothing["Prediction Smoothing"]
end

subgraph group_custom["Custom Model Workflow"]
  node_datasetpage["Dataset Page<br/>[DatasetPage.tsx]"]
  node_trainpage["Train Page<br/>[TrainPage.tsx]"]
  node_evaluatepage["Evaluate Page<br/>[EvaluatePage.tsx]"]
  node_datasetutils["Dataset Processing<br/>[datasetUtils.ts]"]
  node_training["TF.js Model Engine<br/>[modelEngine.ts]"]
  node_model["Model State API<br/>[model.ts]"]
  node_metrics["Model Metrics<br/>[modelEngine.ts]"]
end

subgraph group_state["Browser State"]
  node_context["App State<br/>[AppContext.tsx]"]
  node_datasetstore[("Dataset Storage<br/>[datasetStorage.ts]")]
end

subgraph group_speech["Speech Output"]
  node_tts["Speech Selection<br/>[tts.ts]"]
  node_localtts["Local Speech<br/>[espeakFallback.ts]"]
  node_cloudtts["Cloud TTS Client<br/>[cloudTts.ts]"]
  node_handler["TTS Proxy<br/>[handler.ts]"]
end

node_user(("User"))
node_provider["Google TTS"]

node_user -->|"uses"| node_app
node_app -->|"routes"| node_recognize
node_app -->|"routes"| node_datasetpage
node_app -->|"routes"| node_trainpage
node_app -->|"routes"| node_evaluatepage
node_app -->|"loads model"| node_onnx
node_recognize -->|"starts tracking"| node_mediapipe
node_mediapipe -->|"normalizes"| node_landmarks
node_mediapipe -->|"detects proximity"| node_face
node_landmarks -->|"provides features"| node_onnx
node_onnx -->|"segments predictions"| node_fsm
node_recognize -->|"routes predictions"| node_hybrid
node_hybrid -->|"uses fallback"| node_onnx
node_recognize -->|"confirms predictions"| node_smoothing
node_datasetpage -->|"edits dataset"| node_context
node_trainpage -->|"uses dataset"| node_context
node_evaluatepage -->|"reads metrics"| node_context
node_context -->|"reads and writes"| node_datasetstore
node_context -->|"subscribes to state"| node_model
node_datasetpage -->|"imports samples"| node_datasetutils
node_trainpage -->|"trains or loads"| node_model
node_model -->|"delegates model work"| node_training
node_training -->|"encodes samples"| node_datasetutils
node_training -->|"computes metrics"| node_metrics
node_training -->|"stores model locally"| node_user
node_recognize -->|"requests speech"| node_tts
node_tts -->|"speaks locally"| node_localtts
node_tts -.->|"optionally requests"| node_cloudtts
node_cloudtts -.->|"sends text"| node_handler
node_handler -.->|"requests synthesis"| node_provider

click node_app "https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/App.tsx"
click node_recognize "https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/pages/RecognizePage.tsx"
click node_datasetpage "https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/pages/DatasetPage.tsx"
click node_trainpage "https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/pages/TrainPage.tsx"
click node_evaluatepage "https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/pages/EvaluatePage.tsx"
click node_mediapipe "https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/hooks/useMediaPipe.ts"
click node_landmarks "https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/utils/landmarks.ts"
click node_onnx "https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/ml/inferenceModel.ts"
click node_fsm "https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/ml/segmentationFSM.ts"
click node_face "https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/hooks/useMediaPipe.ts"
click node_hybrid "https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/ml/hybridRouter.ts"
click node_smoothing "https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/hooks/usePredictionSmoothing.ts"
click node_context "https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/contexts/AppContext.tsx"
click node_datasetutils "https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/dataset/datasetUtils.ts"
click node_datasetstore "https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/dataset/datasetStorage.ts"
click node_training "https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/ml/modelEngine.ts"
click node_model "https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/ml/model.ts"
click node_metrics "https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/ml/modelEngine.ts"
click node_tts "https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/utils/tts.ts"
click node_localtts "https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/utils/espeakFallback.ts"
click node_cloudtts "https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/utils/cloudTts.ts"
click node_handler "https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/server/tts/handler.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app,node_user toneBlue
class node_recognize,node_mediapipe,node_landmarks,node_onnx,node_fsm,node_face,node_hybrid,node_smoothing toneAmber
class node_datasetpage,node_trainpage,node_evaluatepage,node_datasetutils,node_training,node_model,node_metrics toneMint
class node_context,node_datasetstore toneRose
class node_tts,node_localtts,node_cloudtts,node_handler,node_provider toneIndigo
```

## Components

### Application

- [App and Routes](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/App.tsx): `src/App.tsx`

### Live Recognition

- [Recognize Page](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/pages/RecognizePage.tsx): `src/pages/RecognizePage.tsx`
- [Camera Tracking](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/hooks/useMediaPipe.ts): `src/hooks/useMediaPipe.ts`
- [Landmark Features](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/utils/landmarks.ts): `src/utils/landmarks.ts`
- [ONNX Inference](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/ml/inferenceModel.ts): `src/ml/inferenceModel.ts`
- [Gesture Segmentation](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/ml/segmentationFSM.ts): `src/ml/segmentationFSM.ts`
- [Face Interactions](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/hooks/useMediaPipe.ts): `src/hooks/useMediaPipe.ts`
- [Hybrid Router](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/ml/hybridRouter.ts): `src/ml/hybridRouter.ts`
- [Prediction Smoothing](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/hooks/usePredictionSmoothing.ts): `src/hooks/usePredictionSmoothing.ts`

### Custom Model Workflow

- [Dataset Page](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/pages/DatasetPage.tsx): `src/pages/DatasetPage.tsx`
- [Train Page](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/pages/TrainPage.tsx): `src/pages/TrainPage.tsx`
- [Evaluate Page](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/pages/EvaluatePage.tsx): `src/pages/EvaluatePage.tsx`
- [Dataset Processing](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/dataset/datasetUtils.ts): `src/dataset/datasetUtils.ts`
- [TF.js Model Engine](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/ml/modelEngine.ts): `src/ml/modelEngine.ts`
- [Model State API](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/ml/model.ts): `src/ml/model.ts`
- [Model Metrics](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/ml/modelEngine.ts): `src/ml/modelEngine.ts`

### Browser State

- [App State](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/contexts/AppContext.tsx): `src/contexts/AppContext.tsx`
- [Dataset Storage](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/dataset/datasetStorage.ts): `src/dataset/datasetStorage.ts`

### Speech Output

- [Speech Selection](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/utils/tts.ts): `src/utils/tts.ts`
- [Local Speech](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/utils/espeakFallback.ts): `src/utils/espeakFallback.ts`
- [Cloud TTS Client](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/src/utils/cloudTts.ts): `src/utils/cloudTts.ts`
- [TTS Proxy](https://github.com/makeitezzz555/signai-asl-gesture-demo/blob/main/server/tts/handler.ts): `server/tts/handler.ts`

### Other components

- **User**
- **Google TTS**

## Connections

- User → App and Routes: uses
- App and Routes → Recognize Page: routes
- App and Routes → Dataset Page: routes
- App and Routes → Train Page: routes
- App and Routes → Evaluate Page: routes
- App and Routes → ONNX Inference: loads model
- Recognize Page → Camera Tracking: starts tracking
- Camera Tracking → Landmark Features: normalizes
- Camera Tracking → Face Interactions: detects proximity
- Landmark Features → ONNX Inference: provides features
- ONNX Inference → Gesture Segmentation: segments predictions
- Recognize Page → Hybrid Router: routes predictions
- Hybrid Router → ONNX Inference: uses fallback
- Recognize Page → Prediction Smoothing: confirms predictions
- Dataset Page → App State: edits dataset
- Train Page → App State: uses dataset
- Evaluate Page → App State: reads metrics
- App State → Dataset Storage: reads and writes
- App State → Model State API: subscribes to state
- Dataset Page → Dataset Processing: imports samples
- Train Page → Model State API: trains or loads
- Model State API → TF.js Model Engine: delegates model work
- TF.js Model Engine → Dataset Processing: encodes samples
- TF.js Model Engine → Model Metrics: computes metrics
- TF.js Model Engine → User: stores model locally
- Recognize Page → Speech Selection: requests speech
- Speech Selection → Local Speech: speaks locally
- Speech Selection → Cloud TTS Client: optionally requests
- Cloud TTS Client → TTS Proxy: sends text
- TTS Proxy → Google TTS: requests synthesis