How richmondbh/voicepilot works
VoicePilot accepts typed or recorded desktop requests, transcribes audio, interprets intent, and routes simple commands through Python validation or complex/document requests through a bounded agent. The UI also exposes history, memories, document indexing/search, and labelled voice-sample recording. The README documents these capabilities; some endpoint and component internals were not sampled, so the graph omits uncertain implementation links.
Main entry points: main.py FastAPI routes · rag.py Document search · agent.py Bounded agent · actions.py Action validation
Components 20
Web interface
- Assistant UI
frontend/src/App.tsx - Voice recorder
frontend/src/components/VoiceRecorder.tsx - Media recorder hook
frontend/src/services/useRecorder.ts - Dataset recorder UI
frontend/src/components/DatasetRecorder.tsx - API client
frontend/src/services/api.ts
Request processing
- FastAPI routes
backend/app/main.py - Speech transcription
backend/app/speech.py - Intent understanding
backend/app/assistant.py - Offline fallback
backend/app/fallback.py - Bounded agent
backend/app/agent.py - Request models
backend/app/models.py
Assistant capabilities
- Action validation
backend/app/actions.py - Document search
backend/app/rag.py
Persistence and data
- User documents
documents - Memory and history
backend/app/memory.py - Search index
backend/app/rag.py - Recorded commands
data/raw/recorded_commands.csv
Other components
- User
- Desktop and browser
- Ollama services
Connections 23
- User → Assistant UI: submits request
README.md - Assistant UI → Voice recorder: renders
frontend/src/App.tsx - Voice recorder → Media recorder hook: records audio
frontend/src/components/VoiceRecorder.tsx - Assistant UI → API client: calls API
frontend/src/App.tsx - Dataset recorder UI → API client: uploads samples
frontend/src/components/DatasetRecorder.tsx - API client → FastAPI routes: sends requests
frontend/src/services/api.ts - FastAPI routes → Speech transcription: transcribes audio
backend/app/main.py - FastAPI routes → Intent understanding: interprets command
backend/app/main.py - Intent understanding → Offline fallback: falls back
backend/app/assistant.py - FastAPI routes → Bounded agent: routes complex requests
backend/app/main.py - FastAPI routes → Action validation: executes validated action
backend/app/main.py - Bounded agent → Action validation: validates tool actions
backend/app/agent.py - Bounded agent → Document search: searches documents
README.md - FastAPI routes → Document search: lists and reindexes
backend/app/main.py - Document search → User documents: reads and indexes
README.md - Document search → Search index: stores index
README.md - FastAPI routes → Memory and history: reads and records
backend/app/main.py - Action validation → Memory and history: saves memory
backend/app/actions.py - Bounded agent → Memory and history: recalls memory
backend/app/agent.py - FastAPI routes → Recorded commands: records labelled samples
README.md - Action validation → Desktop and browser: launches approved targets
backend/app/actions.py - Intent understanding → Ollama services: queries language model
- Document search → Ollama services: requests embeddings
backend/app/rag.py