# richmondbh/voicepilot architecture

> Architecture diagram of the GitHub repository richmondbh/voicepilot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/richmondbh/voicepilot
- Repository: https://github.com/richmondbh/voicepilot

## Overview

VoicePilot accepts typed or recorded desktop requests, transcribes audio, interprets intent, and routes simple commands through Python validation or complex/document requests through a bounded agent. The UI also exposes history, memories, document indexing/search, and labelled voice-sample recording. The README documents these capabilities; some endpoint and component internals were not sampled, so the graph omits uncertain implementation links.

Main entry points: [`backend/app/main.py`](https://github.com/richmondbh/voicepilot/blob/main/backend/app/main.py) (FastAPI routes), [`backend/app/rag.py`](https://github.com/richmondbh/voicepilot/blob/main/backend/app/rag.py) (Document search), [`backend/app/agent.py`](https://github.com/richmondbh/voicepilot/blob/main/backend/app/agent.py) (Bounded agent), [`backend/app/actions.py`](https://github.com/richmondbh/voicepilot/blob/main/backend/app/actions.py) (Action validation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_client["Web interface"]
  node_app_ui["Assistant UI<br/>[App.tsx]"]
  node_voice_recorder["Voice recorder<br/>[VoiceRecorder.tsx]"]
  node_recorder_hook["Media recorder hook<br/>[useRecorder.ts]"]
  node_dataset_ui["Dataset recorder UI"]
  node_api_client["API client<br/>[api.ts]"]
end

subgraph group_orchestration["Request processing"]
  node_api_server["FastAPI routes<br/>[main.py]"]
  node_speech["Speech transcription<br/>[speech.py]"]
  node_intent["Intent understanding<br/>[assistant.py]"]
  node_offline["Offline fallback<br/>[fallback.py]"]
  node_agent["Bounded agent<br/>[agent.py]"]
  node_models["Request models<br/>[models.py]"]
end

subgraph group_capabilities["Assistant capabilities"]
  node_actions["Action validation<br/>[actions.py]"]
  node_rag["Document search<br/>[rag.py]"]
end

subgraph group_state["Persistence and data"]
  node_documents["User documents"]
  node_memories[("Memory and history<br/>[memory.py]")]
  node_rag_index["Search index<br/>[rag.py]"]
  node_voice_samples[("Recorded commands")]
end

node_user(("User"))
node_desktop["Desktop and browser"]
node_ollama["Ollama services"]

node_user -->|"submits request"| node_app_ui
node_app_ui -->|"renders"| node_voice_recorder
node_voice_recorder -->|"records audio"| node_recorder_hook
node_app_ui -->|"calls API"| node_api_client
node_dataset_ui -->|"uploads samples"| node_api_client
node_api_client -->|"sends requests"| node_api_server
node_api_server -->|"transcribes audio"| node_speech
node_api_server -->|"interprets command"| node_intent
node_intent -->|"falls back"| node_offline
node_api_server -->|"routes complex requests"| node_agent
node_api_server -->|"executes validated action"| node_actions
node_agent -->|"validates tool actions"| node_actions
node_agent -->|"searches documents"| node_rag
node_api_server -->|"lists and reindexes"| node_rag
node_rag -->|"reads and indexes"| node_documents
node_rag -->|"stores index"| node_rag_index
node_api_server -->|"reads and records"| node_memories
node_actions -->|"saves memory"| node_memories
node_agent -->|"recalls memory"| node_memories
node_api_server -->|"records labelled samples"| node_voice_samples
node_actions -->|"launches approved targets"| node_desktop
node_intent -.->|"queries language model"| node_ollama
node_rag -.->|"requests embeddings"| node_ollama

click node_app_ui "https://github.com/richmondbh/voicepilot/blob/main/frontend/src/App.tsx"
click node_voice_recorder "https://github.com/richmondbh/voicepilot/blob/main/frontend/src/components/VoiceRecorder.tsx"
click node_recorder_hook "https://github.com/richmondbh/voicepilot/blob/main/frontend/src/services/useRecorder.ts"
click node_dataset_ui "https://github.com/richmondbh/voicepilot/blob/main/frontend/src/components/DatasetRecorder.tsx"
click node_api_client "https://github.com/richmondbh/voicepilot/blob/main/frontend/src/services/api.ts"
click node_api_server "https://github.com/richmondbh/voicepilot/blob/main/backend/app/main.py"
click node_speech "https://github.com/richmondbh/voicepilot/blob/main/backend/app/speech.py"
click node_intent "https://github.com/richmondbh/voicepilot/blob/main/backend/app/assistant.py"
click node_offline "https://github.com/richmondbh/voicepilot/blob/main/backend/app/fallback.py"
click node_agent "https://github.com/richmondbh/voicepilot/blob/main/backend/app/agent.py"
click node_models "https://github.com/richmondbh/voicepilot/blob/main/backend/app/models.py"
click node_actions "https://github.com/richmondbh/voicepilot/blob/main/backend/app/actions.py"
click node_rag "https://github.com/richmondbh/voicepilot/blob/main/backend/app/rag.py"
click node_documents "https://github.com/richmondbh/voicepilot/tree/main/documents"
click node_memories "https://github.com/richmondbh/voicepilot/blob/main/backend/app/memory.py"
click node_rag_index "https://github.com/richmondbh/voicepilot/blob/main/backend/app/rag.py"
click node_voice_samples "https://github.com/richmondbh/voicepilot/blob/main/data/raw/recorded_commands.csv"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app_ui,node_voice_recorder,node_recorder_hook,node_dataset_ui,node_api_client,node_user,node_desktop toneBlue
class node_api_server,node_speech,node_intent,node_offline,node_agent,node_models toneAmber
class node_actions,node_rag toneMint
class node_documents,node_memories,node_rag_index,node_voice_samples toneRose
class node_ollama toneIndigo
```

## Components

### Web interface

- [Assistant UI](https://github.com/richmondbh/voicepilot/blob/main/frontend/src/App.tsx): `frontend/src/App.tsx`
- [Voice recorder](https://github.com/richmondbh/voicepilot/blob/main/frontend/src/components/VoiceRecorder.tsx): `frontend/src/components/VoiceRecorder.tsx`
- [Media recorder hook](https://github.com/richmondbh/voicepilot/blob/main/frontend/src/services/useRecorder.ts): `frontend/src/services/useRecorder.ts`
- [Dataset recorder UI](https://github.com/richmondbh/voicepilot/blob/main/frontend/src/components/DatasetRecorder.tsx): `frontend/src/components/DatasetRecorder.tsx`
- [API client](https://github.com/richmondbh/voicepilot/blob/main/frontend/src/services/api.ts): `frontend/src/services/api.ts`

### Request processing

- [FastAPI routes](https://github.com/richmondbh/voicepilot/blob/main/backend/app/main.py): `backend/app/main.py`
- [Speech transcription](https://github.com/richmondbh/voicepilot/blob/main/backend/app/speech.py): `backend/app/speech.py`
- [Intent understanding](https://github.com/richmondbh/voicepilot/blob/main/backend/app/assistant.py): `backend/app/assistant.py`
- [Offline fallback](https://github.com/richmondbh/voicepilot/blob/main/backend/app/fallback.py): `backend/app/fallback.py`
- [Bounded agent](https://github.com/richmondbh/voicepilot/blob/main/backend/app/agent.py): `backend/app/agent.py`
- [Request models](https://github.com/richmondbh/voicepilot/blob/main/backend/app/models.py): `backend/app/models.py`

### Assistant capabilities

- [Action validation](https://github.com/richmondbh/voicepilot/blob/main/backend/app/actions.py): `backend/app/actions.py`
- [Document search](https://github.com/richmondbh/voicepilot/blob/main/backend/app/rag.py): `backend/app/rag.py`

### Persistence and data

- [User documents](https://github.com/richmondbh/voicepilot/tree/main/documents): `documents`
- [Memory and history](https://github.com/richmondbh/voicepilot/blob/main/backend/app/memory.py): `backend/app/memory.py`
- [Search index](https://github.com/richmondbh/voicepilot/blob/main/backend/app/rag.py): `backend/app/rag.py`
- [Recorded commands](https://github.com/richmondbh/voicepilot/blob/main/data/raw/recorded_commands.csv): `data/raw/recorded_commands.csv`

### Other components

- **User**
- **Desktop and browser**
- **Ollama services**

## Connections

- User → Assistant UI: submits request (evidence: `README.md`)
- Assistant UI → Voice recorder: renders (evidence: `frontend/src/App.tsx`)
- Voice recorder → Media recorder hook: records audio (evidence: `frontend/src/components/VoiceRecorder.tsx`)
- Assistant UI → API client: calls API (evidence: `frontend/src/App.tsx`)
- Dataset recorder UI → API client: uploads samples (evidence: `frontend/src/components/DatasetRecorder.tsx`)
- API client → FastAPI routes: sends requests (evidence: `frontend/src/services/api.ts`)
- FastAPI routes → Speech transcription: transcribes audio (evidence: `backend/app/main.py`)
- FastAPI routes → Intent understanding: interprets command (evidence: `backend/app/main.py`)
- Intent understanding → Offline fallback: falls back (evidence: `backend/app/assistant.py`)
- FastAPI routes → Bounded agent: routes complex requests (evidence: `backend/app/main.py`)
- FastAPI routes → Action validation: executes validated action (evidence: `backend/app/main.py`)
- Bounded agent → Action validation: validates tool actions (evidence: `backend/app/agent.py`)
- Bounded agent → Document search: searches documents (evidence: `README.md`)
- FastAPI routes → Document search: lists and reindexes (evidence: `backend/app/main.py`)
- Document search → User documents: reads and indexes (evidence: `README.md`)
- Document search → Search index: stores index (evidence: `README.md`)
- FastAPI routes → Memory and history: reads and records (evidence: `backend/app/main.py`)
- Action validation → Memory and history: saves memory (evidence: `backend/app/actions.py`)
- Bounded agent → Memory and history: recalls memory (evidence: `backend/app/agent.py`)
- FastAPI routes → Recorded commands: records labelled samples (evidence: `README.md`)
- Action validation → Desktop and browser: launches approved targets (evidence: `backend/app/actions.py`)
- Intent understanding → Ollama services: queries language model
- Document search → Ollama services: requests embeddings (evidence: `backend/app/rag.py`)