# hiteshshaw/aeropoint-ai architecture

> Architecture diagram of the GitHub repository hiteshshaw/aeropoint-ai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hiteshshaw/aeropoint-ai
- Repository: https://github.com/hiteshshaw/aeropoint-ai

## Overview

AeroPoint AI is a Windows on-device accessibility assistant: hand gestures and offline speech are interpreted by a central runtime and translated into desktop actions. Its main path combines webcam landmarks, gesture recognition, voice transcription and intent execution; configuration supplies accessibility profiles, while the controller performs cursor, keyboard and system-volume operations. The README also documents dwell clicking, a HUD, battery throttling, dictation, app launching and profile modes. The graph follows shown calls and documented capabilities; unsampled implementation details are not inferred.

Main entry points: [`main.py`](https://github.com/hiteshshaw/aeropoint-ai/blob/main/main.py) (Assistant Runtime), [`voice/speech_engine.py`](https://github.com/hiteshshaw/aeropoint-ai/blob/main/voice/speech_engine.py) (Voice Intents), [`controller/os_actions.py`](https://github.com/hiteshshaw/aeropoint-ai/blob/main/controller/os_actions.py) (System Controller), [`voice/whisper_listener.py`](https://github.com/hiteshshaw/aeropoint-ai/blob/main/voice/whisper_listener.py) (Whisper Listener)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Assistant Runtime"]
  node_main["Assistant Runtime<br/>[main.py]"]
  node_hud["HUD and Dwell<br/>[main.py]"]
  node_profiles["Access Profiles<br/>[config.py]"]
end

subgraph group_vision["Hand Interaction"]
  node_tracker["Hand Tracker<br/>[hand_tracker.py]"]
  node_gestures["Gesture Recognition"]
end

subgraph group_voice["Offline Voice"]
  node_listener["Whisper Listener"]
  node_intent["Voice Intents<br/>[speech_engine.py]"]
end

subgraph group_control["Desktop Control"]
  node_controller["System Controller<br/>[os_actions.py]"]
end

node_user(("User"))
node_camera(("Webcam"))
node_microphone(("Microphone"))
node_desktop(("Windows Desktop"))
node_localmodels["On-Device Models"]

node_user -->|"gestures"| node_camera
node_camera -->|"video frames"| node_main
node_main -->|"processes frames"| node_tracker
node_tracker -->|"returns landmarks"| node_main
node_main -->|"classifies landmarks"| node_gestures
node_user -->|"speaks commands"| node_microphone
node_microphone -->|"audio stream"| node_listener
node_listener -->|"transcribes with"| node_localmodels
node_listener -->|"executes transcript"| node_intent
node_main -->|"starts listener with"| node_intent
node_intent -->|"invokes desktop actions"| node_controller
node_main -->|"dispatches gesture actions"| node_controller
node_controller -->|"controls cursor and keys"| node_desktop
node_controller -->|"sets system volume"| node_desktop
node_main -->|"renders feedback"| node_hud
node_main -->|"reads runtime settings"| node_profiles
node_intent -->|"applies voice profiles"| node_profiles
node_user -->|"starts assistant"| node_main

click node_main "https://github.com/hiteshshaw/aeropoint-ai/blob/main/main.py"
click node_hud "https://github.com/hiteshshaw/aeropoint-ai/blob/main/main.py"
click node_profiles "https://github.com/hiteshshaw/aeropoint-ai/blob/main/config.py"
click node_tracker "https://github.com/hiteshshaw/aeropoint-ai/blob/main/vision/hand_tracker.py"
click node_gestures "https://github.com/hiteshshaw/aeropoint-ai/blob/main/vision/gesture_recognizer.py"
click node_listener "https://github.com/hiteshshaw/aeropoint-ai/blob/main/voice/whisper_listener.py"
click node_intent "https://github.com/hiteshshaw/aeropoint-ai/blob/main/voice/speech_engine.py"
click node_controller "https://github.com/hiteshshaw/aeropoint-ai/blob/main/controller/os_actions.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_hud,node_profiles,node_user toneBlue
class node_tracker,node_gestures toneAmber
class node_listener,node_intent toneMint
class node_controller toneRose
class node_camera,node_microphone,node_desktop,node_localmodels toneIndigo
```

## Components

### Assistant Runtime

- [Assistant Runtime](https://github.com/hiteshshaw/aeropoint-ai/blob/main/main.py): `main.py`
- [HUD and Dwell](https://github.com/hiteshshaw/aeropoint-ai/blob/main/main.py): `main.py`
- [Access Profiles](https://github.com/hiteshshaw/aeropoint-ai/blob/main/config.py): `config.py`

### Hand Interaction

- [Hand Tracker](https://github.com/hiteshshaw/aeropoint-ai/blob/main/vision/hand_tracker.py): `vision/hand_tracker.py`
- [Gesture Recognition](https://github.com/hiteshshaw/aeropoint-ai/blob/main/vision/gesture_recognizer.py): `vision/gesture_recognizer.py`

### Offline Voice

- [Whisper Listener](https://github.com/hiteshshaw/aeropoint-ai/blob/main/voice/whisper_listener.py): `voice/whisper_listener.py`
- [Voice Intents](https://github.com/hiteshshaw/aeropoint-ai/blob/main/voice/speech_engine.py): `voice/speech_engine.py`

### Desktop Control

- [System Controller](https://github.com/hiteshshaw/aeropoint-ai/blob/main/controller/os_actions.py): `controller/os_actions.py`

### Other components

- **User**
- **Webcam**
- **Microphone**
- **Windows Desktop**
- **On-Device Models**

## Connections

- User → Webcam: gestures (evidence: `README.md`)
- Webcam → Assistant Runtime: video frames (evidence: `main.py`)
- Assistant Runtime → Hand Tracker: processes frames (evidence: `main.py`)
- Hand Tracker → Assistant Runtime: returns landmarks (evidence: `vision/hand_tracker.py`)
- Assistant Runtime → Gesture Recognition: classifies landmarks (evidence: `main.py`)
- User → Microphone: speaks commands (evidence: `README.md`)
- Microphone → Whisper Listener: audio stream (evidence: `voice/whisper_listener.py`)
- Whisper Listener → On-Device Models: transcribes with (evidence: `voice/whisper_listener.py`)
- Whisper Listener → Voice Intents: executes transcript (evidence: `voice/whisper_listener.py`)
- Assistant Runtime → Voice Intents: starts listener with (evidence: `main.py`)
- Voice Intents → System Controller: invokes desktop actions (evidence: `voice/speech_engine.py`)
- Assistant Runtime → System Controller: dispatches gesture actions (evidence: `main.py`)
- System Controller → Windows Desktop: controls cursor and keys (evidence: `controller/os_actions.py`)
- System Controller → Windows Desktop: sets system volume (evidence: `controller/os_actions.py`)
- Assistant Runtime → HUD and Dwell: renders feedback (evidence: `README.md`)
- Assistant Runtime → Access Profiles: reads runtime settings (evidence: `main.py`)
- Voice Intents → Access Profiles: applies voice profiles (evidence: `voice/speech_engine.py`)
- User → Assistant Runtime: starts assistant (evidence: `README.md`)