How vaishnaviii03/umi works
UMI is a personal multimodal assistant spanning a Next.js interface, FastAPI backend, and optional Electron desktop shell. The main interaction runs from chat input through backend orchestration and LLM reasoning; authorized tool requests execute through the tool manager, while relevant history and memories can inform replies. Users also manage tasks and Gmail, Calendar, and Google services; voice adds speech recognition and local synthesis. The desktop shell starts the app, and the background launcher can activate it by double clap. Several integrations are documented in the README but their wiring was not sampled, so their internal links are omitted.
Main entry points: routes.py Backend API · core.py Conversation orchestrator · useChat.ts Chat and voice hooks · useVoiceSession.ts Voice session
Components 24
User interfaces
- UMI interface
frontend/app/UmiInterface.tsx - Chat and voice hooks
frontend/app/hooks/useChat.ts - Voice session
frontend/app/hooks/useVoiceSession.ts - Tasks panel
frontend/app/TasksPanel.tsx - Speech recognition
frontend/app/lib/voice
Assistant backend
- Backend API
backend/app/api/routes.py - API application
backend/app/main.py - Conversation orchestrator
backend/app/orchestrator/core.py - LLM provider manager
backend/app/llm/manager.py - Tool authorization and execution
backend/app/tools/manager.py - Tool implementations
backend/app/tools - Task API
backend/app/api/routes.py - Local speech synthesis
backend/app/services/local_tts.py
Persistence and context
- Memory context
backend/app/services/knowledge_service.py - Conversation and memory access
backend/app/db/repositories.py - Persistent database
backend/app/db/engine.py
Product integrations
- Gmail services
backend/app/services/gmail - Calendar services
backend/app/services/calendar - Drive Sheets YouTube
backend/app/services - Vision service
backend/app/services/vision.py
Desktop activation
- Electron shell
desktop/index.js - Startup orchestration
desktop/startup/orchestrator.js - Double-clap launcher
launcher/main.py
Other components
- User
Connections 20
- User → UMI interface: interacts with
- UMI interface → Chat and voice hooks: handles chat
- Chat and voice hooks → Backend API: sends chat
- Voice session → Speech recognition: selects recognizer
- Chat and voice hooks → Voice session: starts voice
- Backend API → Conversation orchestrator: dispatches turn
- Conversation orchestrator → Memory context: builds context
- Conversation orchestrator → LLM provider manager: requests reasoning
- Conversation orchestrator → Tool authorization and execution: dispatches tools
- Tool authorization and execution → Tool implementations: executes validated tool
- Memory context → Conversation and memory access: retrieves memories
- Conversation and memory access → Persistent database: reads and writes
- Backend API → Conversation and memory access: serves conversations
- Tasks panel → Task API: manages tasks
- Task API → Persistent database: persists task data
- Backend API → Local speech synthesis: synthesizes speech
- Voice session → Backend API: requests STT token
- API application → Backend API: mounts routes
- Electron shell → Startup orchestration: starts services
- Double-clap launcher → Electron shell: signals activation