# vaishnaviii03/umi architecture

> Architecture diagram of the GitHub repository vaishnaviii03/umi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vaishnaviii03/umi
- Repository: https://github.com/vaishnaviii03/umi

## Overview

UMI is a personal multimodal assistant spanning a Next.js interface, FastAPI backend, and optional Electron desktop shell. The main interaction runs from chat input through backend orchestration and LLM reasoning; authorized tool requests execute through the tool manager, while relevant history and memories can inform replies. Users also manage tasks and Gmail, Calendar, and Google services; voice adds speech recognition and local synthesis. The desktop shell starts the app, and the background launcher can activate it by double clap. Several integrations are documented in the README but their wiring was not sampled, so their internal links are omitted.

Main entry points: [`backend/app/api/routes.py`](https://github.com/vaishnaviii03/umi/blob/main/backend/app/api/routes.py) (Backend API), [`backend/app/orchestrator/core.py`](https://github.com/vaishnaviii03/umi/blob/main/backend/app/orchestrator/core.py) (Conversation orchestrator), [`frontend/app/hooks/useChat.ts`](https://github.com/vaishnaviii03/umi/blob/main/frontend/app/hooks/useChat.ts) (Chat and voice hooks), [`frontend/app/hooks/useVoiceSession.ts`](https://github.com/vaishnaviii03/umi/blob/main/frontend/app/hooks/useVoiceSession.ts) (Voice session)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_client["User interfaces"]
  node_umi_ui["UMI interface<br/>[UmiInterface.tsx]"]
  node_chat_hook["Chat and voice hooks<br/>[useChat.ts]"]
  node_voice_session["Voice session<br/>[useVoiceSession.ts]"]
  node_tasks_ui["Tasks panel<br/>[TasksPanel.tsx]"]
  node_speech["Speech recognition"]
end

subgraph group_backend["Assistant backend"]
  node_api["Backend API<br/>[routes.py]"]
  node_app["API application<br/>[main.py]"]
  node_orchestrator["Conversation orchestrator<br/>[core.py]"]
  node_llm["LLM provider manager<br/>[manager.py]"]
  node_tool_manager["Tool authorization and execution<br/>[manager.py]"]
  node_tool_catalog["Tool implementations"]
  node_tasks_api["Task API<br/>[routes.py]"]
  node_tts["Local speech synthesis<br/>[local_tts.py]"]
end

subgraph group_memory["Persistence and context"]
  node_knowledge["Memory context"]
  node_repositories["Conversation and memory access<br/>[repositories.py]"]
  node_database[("Persistent database<br/>[engine.py]")]
end

subgraph group_integrations["Product integrations"]
  node_gmail["Gmail services"]
  node_calendar["Calendar services"]
  node_google_services["Drive Sheets YouTube"]
  node_vision["Vision service<br/>[vision.py]"]
end

subgraph group_desktop["Desktop activation"]
  node_desktop["Electron shell<br/>[index.js]"]
  node_startup["Startup orchestration<br/>[orchestrator.js]"]
  node_launcher["Double-clap launcher<br/>[main.py]"]
end

node_user(("User"))

node_user -->|"interacts with"| node_umi_ui
node_umi_ui -->|"handles chat"| node_chat_hook
node_chat_hook -->|"sends chat"| node_api
node_voice_session -->|"selects recognizer"| node_speech
node_chat_hook -->|"starts voice"| node_voice_session
node_api -->|"dispatches turn"| node_orchestrator
node_orchestrator -->|"builds context"| node_knowledge
node_orchestrator -->|"requests reasoning"| node_llm
node_orchestrator -->|"dispatches tools"| node_tool_manager
node_tool_manager -->|"executes validated tool"| node_tool_catalog
node_knowledge -->|"retrieves memories"| node_repositories
node_repositories -->|"reads and writes"| node_database
node_api -->|"serves conversations"| node_repositories
node_tasks_ui -->|"manages tasks"| node_tasks_api
node_tasks_api -->|"persists task data"| node_database
node_api -->|"synthesizes speech"| node_tts
node_voice_session -->|"requests STT token"| node_api
node_app -->|"mounts routes"| node_api
node_desktop -->|"starts services"| node_startup
node_launcher -->|"signals activation"| node_desktop

click node_umi_ui "https://github.com/vaishnaviii03/umi/blob/main/frontend/app/UmiInterface.tsx"
click node_chat_hook "https://github.com/vaishnaviii03/umi/blob/main/frontend/app/hooks/useChat.ts"
click node_voice_session "https://github.com/vaishnaviii03/umi/blob/main/frontend/app/hooks/useVoiceSession.ts"
click node_tasks_ui "https://github.com/vaishnaviii03/umi/blob/main/frontend/app/TasksPanel.tsx"
click node_api "https://github.com/vaishnaviii03/umi/blob/main/backend/app/api/routes.py"
click node_app "https://github.com/vaishnaviii03/umi/blob/main/backend/app/main.py"
click node_orchestrator "https://github.com/vaishnaviii03/umi/blob/main/backend/app/orchestrator/core.py"
click node_llm "https://github.com/vaishnaviii03/umi/blob/main/backend/app/llm/manager.py"
click node_tool_manager "https://github.com/vaishnaviii03/umi/blob/main/backend/app/tools/manager.py"
click node_tool_catalog "https://github.com/vaishnaviii03/umi/tree/main/backend/app/tools"
click node_knowledge "https://github.com/vaishnaviii03/umi/blob/main/backend/app/services/knowledge_service.py"
click node_repositories "https://github.com/vaishnaviii03/umi/blob/main/backend/app/db/repositories.py"
click node_database "https://github.com/vaishnaviii03/umi/blob/main/backend/app/db/engine.py"
click node_tasks_api "https://github.com/vaishnaviii03/umi/blob/main/backend/app/api/routes.py"
click node_gmail "https://github.com/vaishnaviii03/umi/tree/main/backend/app/services/gmail"
click node_calendar "https://github.com/vaishnaviii03/umi/tree/main/backend/app/services/calendar"
click node_google_services "https://github.com/vaishnaviii03/umi/tree/main/backend/app/services"
click node_vision "https://github.com/vaishnaviii03/umi/blob/main/backend/app/services/vision.py"
click node_speech "https://github.com/vaishnaviii03/umi/tree/main/frontend/app/lib/voice"
click node_tts "https://github.com/vaishnaviii03/umi/blob/main/backend/app/services/local_tts.py"
click node_desktop "https://github.com/vaishnaviii03/umi/blob/main/desktop/index.js"
click node_startup "https://github.com/vaishnaviii03/umi/blob/main/desktop/startup/orchestrator.js"
click node_launcher "https://github.com/vaishnaviii03/umi/blob/main/launcher/main.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_umi_ui,node_chat_hook,node_voice_session,node_tasks_ui,node_speech,node_user toneBlue
class node_api,node_app,node_orchestrator,node_llm,node_tool_manager,node_tool_catalog,node_tasks_api,node_tts toneAmber
class node_knowledge,node_repositories,node_database toneMint
class node_gmail,node_calendar,node_google_services,node_vision toneRose
class node_desktop,node_startup,node_launcher toneIndigo
```

## Components

### User interfaces

- [UMI interface](https://github.com/vaishnaviii03/umi/blob/main/frontend/app/UmiInterface.tsx): `frontend/app/UmiInterface.tsx`
- [Chat and voice hooks](https://github.com/vaishnaviii03/umi/blob/main/frontend/app/hooks/useChat.ts): `frontend/app/hooks/useChat.ts`
- [Voice session](https://github.com/vaishnaviii03/umi/blob/main/frontend/app/hooks/useVoiceSession.ts): `frontend/app/hooks/useVoiceSession.ts`
- [Tasks panel](https://github.com/vaishnaviii03/umi/blob/main/frontend/app/TasksPanel.tsx): `frontend/app/TasksPanel.tsx`
- [Speech recognition](https://github.com/vaishnaviii03/umi/tree/main/frontend/app/lib/voice): `frontend/app/lib/voice`

### Assistant backend

- [Backend API](https://github.com/vaishnaviii03/umi/blob/main/backend/app/api/routes.py): `backend/app/api/routes.py`
- [API application](https://github.com/vaishnaviii03/umi/blob/main/backend/app/main.py): `backend/app/main.py`
- [Conversation orchestrator](https://github.com/vaishnaviii03/umi/blob/main/backend/app/orchestrator/core.py): `backend/app/orchestrator/core.py`
- [LLM provider manager](https://github.com/vaishnaviii03/umi/blob/main/backend/app/llm/manager.py): `backend/app/llm/manager.py`
- [Tool authorization and execution](https://github.com/vaishnaviii03/umi/blob/main/backend/app/tools/manager.py): `backend/app/tools/manager.py`
- [Tool implementations](https://github.com/vaishnaviii03/umi/tree/main/backend/app/tools): `backend/app/tools`
- [Task API](https://github.com/vaishnaviii03/umi/blob/main/backend/app/api/routes.py): `backend/app/api/routes.py`
- [Local speech synthesis](https://github.com/vaishnaviii03/umi/blob/main/backend/app/services/local_tts.py): `backend/app/services/local_tts.py`

### Persistence and context

- [Memory context](https://github.com/vaishnaviii03/umi/blob/main/backend/app/services/knowledge_service.py): `backend/app/services/knowledge_service.py`
- [Conversation and memory access](https://github.com/vaishnaviii03/umi/blob/main/backend/app/db/repositories.py): `backend/app/db/repositories.py`
- [Persistent database](https://github.com/vaishnaviii03/umi/blob/main/backend/app/db/engine.py): `backend/app/db/engine.py`

### Product integrations

- [Gmail services](https://github.com/vaishnaviii03/umi/tree/main/backend/app/services/gmail): `backend/app/services/gmail`
- [Calendar services](https://github.com/vaishnaviii03/umi/tree/main/backend/app/services/calendar): `backend/app/services/calendar`
- [Drive Sheets YouTube](https://github.com/vaishnaviii03/umi/tree/main/backend/app/services): `backend/app/services`
- [Vision service](https://github.com/vaishnaviii03/umi/blob/main/backend/app/services/vision.py): `backend/app/services/vision.py`

### Desktop activation

- [Electron shell](https://github.com/vaishnaviii03/umi/blob/main/desktop/index.js): `desktop/index.js`
- [Startup orchestration](https://github.com/vaishnaviii03/umi/blob/main/desktop/startup/orchestrator.js): `desktop/startup/orchestrator.js`
- [Double-clap launcher](https://github.com/vaishnaviii03/umi/blob/main/launcher/main.py): `launcher/main.py`

### Other components

- **User**

## Connections

- User → UMI interface: interacts with
- UMI interface → Chat and voice hooks: handles chat
- Chat and voice hooks → Backend API: sends chat
- Voice session → Speech recognition: selects recognizer
- Chat and voice hooks → Voice session: starts voice
- Backend API → Conversation orchestrator: dispatches turn
- Conversation orchestrator → Memory context: builds context
- Conversation orchestrator → LLM provider manager: requests reasoning
- Conversation orchestrator → Tool authorization and execution: dispatches tools
- Tool authorization and execution → Tool implementations: executes validated tool
- Memory context → Conversation and memory access: retrieves memories
- Conversation and memory access → Persistent database: reads and writes
- Backend API → Conversation and memory access: serves conversations
- Tasks panel → Task API: manages tasks
- Task API → Persistent database: persists task data
- Backend API → Local speech synthesis: synthesizes speech
- Voice session → Backend API: requests STT token
- API application → Backend API: mounts routes
- Electron shell → Startup orchestration: starts services
- Double-clap launcher → Electron shell: signals activation