# ishabrato/visual-voice architecture

> Architecture diagram of the GitHub repository ishabrato/visual-voice, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ishabrato/visual-voice
- Repository: https://github.com/ishabrato/visual-voice

## Overview

Visual Voice recognizes ASL fingerspelling from a webcam and turns stable, confident letter/SPACE predictions into editable sentence text. The primary user path is the PyQt UI: capture frames, detect hand landmarks, normalize them, classify gestures, stabilize entries, then display and edit the sentence. Separate OpenCV scripts provide gesture-data logging and classifier demos; the README also describes a notebook-based training workflow. The UI and scripts are excerpted, but some internal setup is omitted, so only documented or directly visible links are drawn.

Main entry points: [`ui.py`](https://github.com/ishabrato/visual-voice/blob/main/ui.py) (Live recognition UI), [`model/keypoint_classifier/keypoint_classifier.py`](https://github.com/ishabrato/visual-voice/blob/main/model/keypoint_classifier/keypoint_classifier.py) (Keypoint classifier), [`app.py`](https://github.com/ishabrato/visual-voice/blob/main/app.py) (Gesture data logging), [`model/point_history_classifier/point_history_classifier.py`](https://github.com/ishabrato/visual-voice/blob/main/model/point_history_classifier/point_history_classifier.py) (Point-history classifier)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_capture["Capture and perception"]
  node_ui["Live recognition UI<br/>[ui.py]"]
  node_landmarks["Hand tracking and landmarks<br/>[ui.py]"]
  node_normalize["Landmark normalization<br/>[ui.py]"]
  node_opencv_demo["OpenCV gesture demo<br/>[app.py]"]
  node_opencv_demo_alt["Alternate gesture demo<br/>[app2.py]"]
end

subgraph group_recognition["Recognition models"]
  node_key_classifier["Keypoint classifier"]
  node_key_model["Keypoint TFLite model"]
  node_history_classifier["Point-history classifier"]
  node_history_model["Point-history TFLite model"]
end

subgraph group_communication["Sentence interface"]
  node_sentence["Stability and sentence builder<br/>[ui.py]"]
  node_sentence_controls["Copy, clear, backspace<br/>[ui.py]"]
  node_display["Video and sentence display<br/>[ui.py]"]
end

subgraph group_extension["Data and training"]
  node_logger["Gesture data logging<br/>[app.py]"]
  node_key_csv[("Keypoint training data<br/>[keypoint.csv]")]
  node_history_csv[("Point-history training data<br/>[point_history.csv]")]
  node_training["Model training notebook"]
end

node_user(("ASL signer"))
node_webcam["Webcam"]

node_user -->|"shows gestures"| node_webcam
node_webcam -->|"provides frames"| node_ui
node_ui -->|"processes frames"| node_landmarks
node_landmarks -->|"extracts coordinates"| node_normalize
node_normalize -->|"feeds landmarks"| node_key_classifier
node_ui -->|"classifies landmarks"| node_key_classifier
node_key_classifier -->|"invokes"| node_key_model
node_key_classifier -->|"returns gesture and confidence"| node_sentence
node_sentence -->|"updates text"| node_display
node_sentence_controls -->|"edits sentence"| node_sentence
node_ui -->|"renders video"| node_display
node_opencv_demo -->|"classifies point history"| node_history_classifier
node_history_classifier -->|"invokes"| node_history_model
node_logger -->|"appends keypoints"| node_key_csv
node_logger -->|"appends history"| node_history_csv
node_key_csv -->|"training input"| node_training
node_training -->|"exports inference model"| node_key_model

click node_ui "https://github.com/ishabrato/visual-voice/blob/main/ui.py"
click node_landmarks "https://github.com/ishabrato/visual-voice/blob/main/ui.py"
click node_normalize "https://github.com/ishabrato/visual-voice/blob/main/ui.py"
click node_key_classifier "https://github.com/ishabrato/visual-voice/blob/main/model/keypoint_classifier/keypoint_classifier.py"
click node_key_model "https://github.com/ishabrato/visual-voice/blob/main/model/keypoint_classifier/keypoint_classifier.tflite"
click node_sentence "https://github.com/ishabrato/visual-voice/blob/main/ui.py"
click node_sentence_controls "https://github.com/ishabrato/visual-voice/blob/main/ui.py"
click node_display "https://github.com/ishabrato/visual-voice/blob/main/ui.py"
click node_opencv_demo "https://github.com/ishabrato/visual-voice/blob/main/app.py"
click node_opencv_demo_alt "https://github.com/ishabrato/visual-voice/blob/main/app2.py"
click node_history_classifier "https://github.com/ishabrato/visual-voice/blob/main/model/point_history_classifier/point_history_classifier.py"
click node_history_model "https://github.com/ishabrato/visual-voice/blob/main/model/point_history_classifier/point_history_classifier.tflite"
click node_logger "https://github.com/ishabrato/visual-voice/blob/main/app.py"
click node_key_csv "https://github.com/ishabrato/visual-voice/blob/main/model/keypoint_classifier/keypoint.csv"
click node_history_csv "https://github.com/ishabrato/visual-voice/blob/main/model/point_history_classifier/point_history.csv"
click node_training "https://github.com/ishabrato/visual-voice/blob/main/keypoint_classification_EN.ipynb"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ui,node_landmarks,node_normalize,node_opencv_demo,node_opencv_demo_alt toneBlue
class node_key_classifier,node_key_model,node_history_classifier,node_history_model toneAmber
class node_sentence,node_sentence_controls,node_display toneMint
class node_logger,node_key_csv,node_history_csv,node_training toneRose
class node_user,node_webcam toneIndigo
```

## Components

### Capture and perception

- [Live recognition UI](https://github.com/ishabrato/visual-voice/blob/main/ui.py): `ui.py`
- [Hand tracking and landmarks](https://github.com/ishabrato/visual-voice/blob/main/ui.py): `ui.py`
- [Landmark normalization](https://github.com/ishabrato/visual-voice/blob/main/ui.py): `ui.py`
- [OpenCV gesture demo](https://github.com/ishabrato/visual-voice/blob/main/app.py): `app.py`
- [Alternate gesture demo](https://github.com/ishabrato/visual-voice/blob/main/app2.py): `app2.py`

### Recognition models

- [Keypoint classifier](https://github.com/ishabrato/visual-voice/blob/main/model/keypoint_classifier/keypoint_classifier.py): `model/keypoint_classifier/keypoint_classifier.py`
- [Keypoint TFLite model](https://github.com/ishabrato/visual-voice/blob/main/model/keypoint_classifier/keypoint_classifier.tflite): `model/keypoint_classifier/keypoint_classifier.tflite`
- [Point-history classifier](https://github.com/ishabrato/visual-voice/blob/main/model/point_history_classifier/point_history_classifier.py): `model/point_history_classifier/point_history_classifier.py`
- [Point-history TFLite model](https://github.com/ishabrato/visual-voice/blob/main/model/point_history_classifier/point_history_classifier.tflite): `model/point_history_classifier/point_history_classifier.tflite`

### Sentence interface

- [Stability and sentence builder](https://github.com/ishabrato/visual-voice/blob/main/ui.py): `ui.py`
- [Copy, clear, backspace](https://github.com/ishabrato/visual-voice/blob/main/ui.py): `ui.py`
- [Video and sentence display](https://github.com/ishabrato/visual-voice/blob/main/ui.py): `ui.py`

### Data and training

- [Gesture data logging](https://github.com/ishabrato/visual-voice/blob/main/app.py): `app.py`
- [Keypoint training data](https://github.com/ishabrato/visual-voice/blob/main/model/keypoint_classifier/keypoint.csv): `model/keypoint_classifier/keypoint.csv`
- [Point-history training data](https://github.com/ishabrato/visual-voice/blob/main/model/point_history_classifier/point_history.csv): `model/point_history_classifier/point_history.csv`
- [Model training notebook](https://github.com/ishabrato/visual-voice/blob/main/keypoint_classification_EN.ipynb): `keypoint_classification_EN.ipynb`

### Other components

- **ASL signer**
- **Webcam**

## Connections

- ASL signer → Webcam: shows gestures (evidence: `README.md`)
- Webcam → Live recognition UI: provides frames (evidence: `ui.py`)
- Live recognition UI → Hand tracking and landmarks: processes frames (evidence: `ui.py`)
- Hand tracking and landmarks → Landmark normalization: extracts coordinates (evidence: `README.md`)
- Landmark normalization → Keypoint classifier: feeds landmarks (evidence: `README.md`)
- Live recognition UI → Keypoint classifier: classifies landmarks (evidence: `ui.py`)
- Keypoint classifier → Keypoint TFLite model: invokes (evidence: `model/keypoint_classifier/keypoint_classifier.py`)
- Keypoint classifier → Stability and sentence builder: returns gesture and confidence (evidence: `ui.py`)
- Stability and sentence builder → Video and sentence display: updates text (evidence: `ui.py`)
- Copy, clear, backspace → Stability and sentence builder: edits sentence (evidence: `ui.py`)
- Live recognition UI → Video and sentence display: renders video (evidence: `ui.py`)
- OpenCV gesture demo → Point-history classifier: classifies point history (evidence: `app.py`)
- Point-history classifier → Point-history TFLite model: invokes (evidence: `model/point_history_classifier/point_history_classifier.py`)
- Gesture data logging → Keypoint training data: appends keypoints (evidence: `app.py`)
- Gesture data logging → Point-history training data: appends history (evidence: `app.py`)
- Keypoint training data → Model training notebook: training input (evidence: `README.md`)
- Model training notebook → Keypoint TFLite model: exports inference model (evidence: `README.md`)
