# daniyaa1/sign2sentence architecture

> Architecture diagram of the GitHub repository daniyaa1/sign2sentence, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/daniyaa1/sign2sentence
- Repository: https://github.com/daniyaa1/sign2sentence

## Overview

AVA provides two webcam inference modes: a single-sign recognizer and a continuous sentence translator. Both capture video, detect hand landmarks, prepare sequences, and classify them with the LSTM; the continuous mode stabilizes recognized glosses and turns them into English using a fast dictionary or Gemini. The README documents these capabilities, while the excerpts show key runtime calls. The continuous-mode sentence dispatch and some asset-loading details are only partially excerpted, so those links rely on documented behavior where noted.

Main entry points: [`predict.py`](https://github.com/daniyaa1/sign2sentence/blob/main/predict.py) (Single-sign mode), [`predict_continuous.py`](https://github.com/daniyaa1/sign2sentence/blob/main/predict_continuous.py) (Continuous mode), [`model/best_model.h5`](https://github.com/daniyaa1/sign2sentence/blob/main/model/best_model.h5) (LSTM classifier), [`output/label_map.npy`](https://github.com/daniyaa1/sign2sentence/blob/main/output/label_map.npy) (Gesture labels)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_capture["Capture and features"]
  node_single["Single-sign mode<br/>[predict.py]"]
  node_continuous["Continuous mode"]
  node_landmarks["Hand tracking<br/>[predict.py]"]
  node_features["Sequence preprocessing<br/>[predict.py]"]
  node_continuous_features["Sequence preprocessing"]
end

subgraph group_recognition["Gesture recognition"]
  node_lstm["LSTM classifier<br/>[best_model.h5]"]
  node_gloss["Gloss stabilization"]
  node_single_display["Prediction display<br/>[predict.py]"]
end

subgraph group_translation["Sentence translation"]
  node_fast_dict["Fast phrase dictionary"]
  node_gemini["Gemini grammar engine"]
  node_sentence_display["Sentence display"]
end

subgraph group_assets["Inference assets"]
  node_labels["Gesture labels<br/>[label_map.npy]"]
  node_scaler["Feature scaler<br/>[scaler.pkl]"]
  node_hand_model["Landmarker model"]
end

node_user(("Signer"))
node_camera(("Webcam"))
node_gemini_service{{"Google Gemini"}}

node_user -->|"signs to"| node_camera
node_camera -->|"provides frames"| node_single
node_camera -->|"provides frames"| node_continuous
node_single -->|"detects hands"| node_landmarks
node_continuous -->|"detects hands"| node_landmarks
node_hand_model -->|"configures"| node_landmarks
node_landmarks -->|"extracts keypoints"| node_features
node_landmarks -->|"extracts keypoints"| node_continuous_features
node_single -->|"preprocesses sequence"| node_features
node_continuous -->|"preprocesses sequence"| node_continuous_features
node_features -->|"scales with"| node_scaler
node_continuous_features -->|"scales with"| node_scaler
node_single -->|"predicts sequence"| node_lstm
node_continuous -->|"predicts sequence"| node_lstm
node_single -->|"maps classes"| node_labels
node_continuous -.->|"maps classes"| node_labels
node_single -->|"shows prediction"| node_single_display
node_continuous -->|"stabilizes words"| node_gloss
node_gloss -->|"looks up phrase"| node_fast_dict
node_gloss -->|"translates unknown"| node_gemini
node_gemini -->|"requests sentence"| node_gemini_service
node_fast_dict -.->|"supplies sentence"| node_sentence_display
node_gemini -.->|"supplies sentence"| node_sentence_display

click node_single "https://github.com/daniyaa1/sign2sentence/blob/main/predict.py"
click node_continuous "https://github.com/daniyaa1/sign2sentence/blob/main/predict_continuous.py"
click node_landmarks "https://github.com/daniyaa1/sign2sentence/blob/main/predict.py"
click node_features "https://github.com/daniyaa1/sign2sentence/blob/main/predict.py"
click node_continuous_features "https://github.com/daniyaa1/sign2sentence/blob/main/predict_continuous.py"
click node_lstm "https://github.com/daniyaa1/sign2sentence/blob/main/model/best_model.h5"
click node_labels "https://github.com/daniyaa1/sign2sentence/blob/main/output/label_map.npy"
click node_scaler "https://github.com/daniyaa1/sign2sentence/blob/main/output/scaler.pkl"
click node_hand_model "https://github.com/daniyaa1/sign2sentence/blob/main/hand_landmarker.task"
click node_gloss "https://github.com/daniyaa1/sign2sentence/blob/main/predict_continuous.py"
click node_fast_dict "https://github.com/daniyaa1/sign2sentence/blob/main/predict_continuous.py"
click node_gemini "https://github.com/daniyaa1/sign2sentence/blob/main/predict_continuous.py"
click node_single_display "https://github.com/daniyaa1/sign2sentence/blob/main/predict.py"
click node_sentence_display "https://github.com/daniyaa1/sign2sentence/blob/main/predict_continuous.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_single,node_continuous,node_landmarks,node_features,node_continuous_features toneBlue
class node_lstm,node_gloss,node_single_display toneAmber
class node_fast_dict,node_gemini,node_sentence_display toneMint
class node_labels,node_scaler,node_hand_model toneRose
class node_user,node_camera,node_gemini_service toneIndigo
```

## Components

### Capture and features

- [Single-sign mode](https://github.com/daniyaa1/sign2sentence/blob/main/predict.py): `predict.py`
- [Continuous mode](https://github.com/daniyaa1/sign2sentence/blob/main/predict_continuous.py): `predict_continuous.py`
- [Hand tracking](https://github.com/daniyaa1/sign2sentence/blob/main/predict.py): `predict.py`
- [Sequence preprocessing](https://github.com/daniyaa1/sign2sentence/blob/main/predict.py): `predict.py`
- [Sequence preprocessing](https://github.com/daniyaa1/sign2sentence/blob/main/predict_continuous.py): `predict_continuous.py`

### Gesture recognition

- [LSTM classifier](https://github.com/daniyaa1/sign2sentence/blob/main/model/best_model.h5): `model/best_model.h5`
- [Gloss stabilization](https://github.com/daniyaa1/sign2sentence/blob/main/predict_continuous.py): `predict_continuous.py`
- [Prediction display](https://github.com/daniyaa1/sign2sentence/blob/main/predict.py): `predict.py`

### Sentence translation

- [Fast phrase dictionary](https://github.com/daniyaa1/sign2sentence/blob/main/predict_continuous.py): `predict_continuous.py`
- [Gemini grammar engine](https://github.com/daniyaa1/sign2sentence/blob/main/predict_continuous.py): `predict_continuous.py`
- [Sentence display](https://github.com/daniyaa1/sign2sentence/blob/main/predict_continuous.py): `predict_continuous.py`

### Inference assets

- [Gesture labels](https://github.com/daniyaa1/sign2sentence/blob/main/output/label_map.npy): `output/label_map.npy`
- [Feature scaler](https://github.com/daniyaa1/sign2sentence/blob/main/output/scaler.pkl): `output/scaler.pkl`
- [Landmarker model](https://github.com/daniyaa1/sign2sentence/blob/main/hand_landmarker.task): `hand_landmarker.task`

### Other components

- **Signer**
- **Webcam**
- **Google Gemini**

## Connections

- Signer → Webcam: signs to (evidence: `README.md`)
- Webcam → Single-sign mode: provides frames (evidence: `predict.py`)
- Webcam → Continuous mode: provides frames (evidence: `predict_continuous.py`)
- Single-sign mode → Hand tracking: detects hands (evidence: `predict.py`)
- Continuous mode → Hand tracking: detects hands (evidence: `predict_continuous.py`)
- Landmarker model → Hand tracking: configures (evidence: `predict.py`)
- Hand tracking → Sequence preprocessing: extracts keypoints (evidence: `predict.py`)
- Hand tracking → Sequence preprocessing: extracts keypoints (evidence: `predict_continuous.py`)
- Single-sign mode → Sequence preprocessing: preprocesses sequence (evidence: `predict.py`)
- Continuous mode → Sequence preprocessing: preprocesses sequence (evidence: `predict_continuous.py`)
- Sequence preprocessing → Feature scaler: scales with (evidence: `predict.py`)
- Sequence preprocessing → Feature scaler: scales with (evidence: `predict_continuous.py`)
- Single-sign mode → LSTM classifier: predicts sequence (evidence: `predict.py`)
- Continuous mode → LSTM classifier: predicts sequence (evidence: `predict_continuous.py`)
- Single-sign mode → Gesture labels: maps classes (evidence: `predict.py`)
- Continuous mode → Gesture labels: maps classes (evidence: `README.md`)
- Single-sign mode → Prediction display: shows prediction (evidence: `predict.py`)
- Continuous mode → Gloss stabilization: stabilizes words (evidence: `predict_continuous.py`)
- Gloss stabilization → Fast phrase dictionary: looks up phrase (evidence: `predict_continuous.py`)
- Gloss stabilization → Gemini grammar engine: translates unknown (evidence: `predict_continuous.py`)
- Gemini grammar engine → Google Gemini: requests sentence (evidence: `predict_continuous.py`)
- Fast phrase dictionary → Sentence display: supplies sentence (evidence: `README.md`)
- Gemini grammar engine → Sentence display: supplies sentence (evidence: `README.md`)
