# thaisontv123/jarvis architecture

> Architecture diagram of the GitHub repository thaisontv123/jarvis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thaisontv123/jarvis
- Repository: https://github.com/thaisontv123/jarvis

## Overview

Jarvis is a Vietnamese, voice-first personal assistant delivered as a Windows desktop app with a local Python API. The main workflow runs from the desktop or chat API through conversation orchestration, model responses, and registered tools; risky actions require permission checks and confirmation before execution. Capability packs cover productivity, web, Google services, and computer control. Separate branches provide persistent memory, scheduled notifications, and user-approved automated routines. The README documents these capabilities; some subsystem connections are shown from documentation because their implementation was not sampled.

Main entry points: [`app/core/orchestrator.py`](https://github.com/thaisontv123/jarvis/blob/master/app/core/orchestrator.py) (Agent orchestration), [`app/api/routes_chat.py`](https://github.com/thaisontv123/jarvis/blob/master/app/api/routes_chat.py) (Chat endpoint), [`app/core/registry.py`](https://github.com/thaisontv123/jarvis/blob/master/app/core/registry.py) (Tool registry), [`app/scheduler/engine.py`](https://github.com/thaisontv123/jarvis/blob/master/app/scheduler/engine.py) (Scheduler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interface["User interfaces"]
  node_desktop["Desktop shell<br/>[main.rs]"]
  node_desktop_ui["Desktop UI<br/>[desktop.js]"]
  node_api["Local API<br/>[main.py]"]
  node_chat["Chat endpoint<br/>[routes_chat.py]"]
end

subgraph group_conversation["Conversation and voice"]
  node_voice["Voice session<br/>[session.py]"]
  node_stt["Speech recognition"]
  node_orchestrator["Agent orchestration<br/>[orchestrator.py]"]
  node_provider["AI provider<br/>[gemini_provider.py]"]
  node_tts["Speech output<br/>[workflow.py]"]
end

subgraph group_capabilities["Capability packs"]
  node_packs["Capability packs<br/>[builtin.py]"]
  node_computer["Computer control<br/>[service.py]"]
  node_integrations["External integrations<br/>[runtime.py]"]
  node_memory["Memory retrieval<br/>[retrieval.py]"]
end

subgraph group_safety["Safety and state"]
  node_registry["Tool registry<br/>[registry.py]"]
  node_permissions["Permission engine<br/>[permissions.py]"]
  node_confirmation["Confirmation<br/>[confirmation.py]"]
  node_audit["Action audit<br/>[audit.py]"]
  node_database[("Application database<br/>[models.py]")]
end

subgraph group_proactive["Proactive automation"]
  node_scheduler["Scheduler<br/>[engine.py]"]
  node_routines["Automated routines<br/>[engine.py]"]
  node_observer["Observations<br/>[observations.py]"]
end

node_user(("User"))

node_user -->|"uses"| node_desktop
node_desktop -->|"opens"| node_desktop_ui
node_desktop_ui -->|"calls same-origin"| node_api
node_api -->|"routes requests"| node_chat
node_user -->|"sends chat"| node_chat
node_chat -->|"invokes"| node_orchestrator
node_voice -.->|"transcribes"| node_stt
node_voice -.->|"submits turns"| node_orchestrator
node_orchestrator -.->|"requests responses"| node_provider
node_orchestrator -->|"dispatches tools"| node_registry
node_registry -->|"checks risk"| node_permissions
node_orchestrator -->|"requests approval"| node_confirmation
node_confirmation -->|"executes approved"| node_registry
node_confirmation -->|"records outcome"| node_audit
node_chat -->|"uses sessions"| node_database
node_registry -.->|"registers capabilities"| node_packs
node_packs -->|"includes computer"| node_computer
node_packs -.->|"uses service clients"| node_integrations
node_orchestrator -.->|"retrieves context"| node_memory
node_scheduler -->|"reads and writes"| node_database
node_scheduler -.->|"runs schedules"| node_routines
node_scheduler -.->|"delivers speech"| node_tts
node_observer -.->|"stores observations"| node_database
node_user -->|"speaks"| node_voice

click node_desktop "https://github.com/thaisontv123/jarvis/blob/master/desktop/src/main.rs"
click node_desktop_ui "https://github.com/thaisontv123/jarvis/blob/master/app/desktop/static/desktop.js"
click node_api "https://github.com/thaisontv123/jarvis/blob/master/app/main.py"
click node_chat "https://github.com/thaisontv123/jarvis/blob/master/app/api/routes_chat.py"
click node_voice "https://github.com/thaisontv123/jarvis/blob/master/app/desktop/session.py"
click node_stt "https://github.com/thaisontv123/jarvis/blob/master/app/stt/gemini_live_provider.py"
click node_orchestrator "https://github.com/thaisontv123/jarvis/blob/master/app/core/orchestrator.py"
click node_provider "https://github.com/thaisontv123/jarvis/blob/master/app/providers/gemini_provider.py"
click node_tts "https://github.com/thaisontv123/jarvis/blob/master/app/tts/workflow.py"
click node_registry "https://github.com/thaisontv123/jarvis/blob/master/app/core/registry.py"
click node_permissions "https://github.com/thaisontv123/jarvis/blob/master/app/core/permissions.py"
click node_confirmation "https://github.com/thaisontv123/jarvis/blob/master/app/core/confirmation.py"
click node_audit "https://github.com/thaisontv123/jarvis/blob/master/app/core/audit.py"
click node_database "https://github.com/thaisontv123/jarvis/blob/master/app/database/models.py"
click node_packs "https://github.com/thaisontv123/jarvis/blob/master/app/packs/builtin.py"
click node_computer "https://github.com/thaisontv123/jarvis/blob/master/app/packs/computer/service.py"
click node_integrations "https://github.com/thaisontv123/jarvis/blob/master/app/integrations/runtime.py"
click node_memory "https://github.com/thaisontv123/jarvis/blob/master/app/memory/retrieval.py"
click node_scheduler "https://github.com/thaisontv123/jarvis/blob/master/app/scheduler/engine.py"
click node_routines "https://github.com/thaisontv123/jarvis/blob/master/app/routines/engine.py"
click node_observer "https://github.com/thaisontv123/jarvis/blob/master/app/memory/observations.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_desktop,node_desktop_ui,node_api,node_chat,node_user toneBlue
class node_voice,node_stt,node_orchestrator,node_provider,node_tts toneAmber
class node_packs,node_computer,node_integrations,node_memory toneMint
class node_registry,node_permissions,node_confirmation,node_audit,node_database toneRose
class node_scheduler,node_routines,node_observer toneIndigo
```

## Components

### User interfaces

- [Desktop shell](https://github.com/thaisontv123/jarvis/blob/master/desktop/src/main.rs): `desktop/src/main.rs`
- [Desktop UI](https://github.com/thaisontv123/jarvis/blob/master/app/desktop/static/desktop.js): `app/desktop/static/desktop.js`
- [Local API](https://github.com/thaisontv123/jarvis/blob/master/app/main.py): `app/main.py`
- [Chat endpoint](https://github.com/thaisontv123/jarvis/blob/master/app/api/routes_chat.py): `app/api/routes_chat.py`

### Conversation and voice

- [Voice session](https://github.com/thaisontv123/jarvis/blob/master/app/desktop/session.py): `app/desktop/session.py`
- [Speech recognition](https://github.com/thaisontv123/jarvis/blob/master/app/stt/gemini_live_provider.py): `app/stt/gemini_live_provider.py`
- [Agent orchestration](https://github.com/thaisontv123/jarvis/blob/master/app/core/orchestrator.py): `app/core/orchestrator.py`
- [AI provider](https://github.com/thaisontv123/jarvis/blob/master/app/providers/gemini_provider.py): `app/providers/gemini_provider.py`
- [Speech output](https://github.com/thaisontv123/jarvis/blob/master/app/tts/workflow.py): `app/tts/workflow.py`

### Capability packs

- [Capability packs](https://github.com/thaisontv123/jarvis/blob/master/app/packs/builtin.py): `app/packs/builtin.py`
- [Computer control](https://github.com/thaisontv123/jarvis/blob/master/app/packs/computer/service.py): `app/packs/computer/service.py`
- [External integrations](https://github.com/thaisontv123/jarvis/blob/master/app/integrations/runtime.py): `app/integrations/runtime.py`
- [Memory retrieval](https://github.com/thaisontv123/jarvis/blob/master/app/memory/retrieval.py): `app/memory/retrieval.py`

### Safety and state

- [Tool registry](https://github.com/thaisontv123/jarvis/blob/master/app/core/registry.py): `app/core/registry.py`
- [Permission engine](https://github.com/thaisontv123/jarvis/blob/master/app/core/permissions.py): `app/core/permissions.py`
- [Confirmation](https://github.com/thaisontv123/jarvis/blob/master/app/core/confirmation.py): `app/core/confirmation.py`
- [Action audit](https://github.com/thaisontv123/jarvis/blob/master/app/core/audit.py): `app/core/audit.py`
- [Application database](https://github.com/thaisontv123/jarvis/blob/master/app/database/models.py): `app/database/models.py`

### Proactive automation

- [Scheduler](https://github.com/thaisontv123/jarvis/blob/master/app/scheduler/engine.py): `app/scheduler/engine.py`
- [Automated routines](https://github.com/thaisontv123/jarvis/blob/master/app/routines/engine.py): `app/routines/engine.py`
- [Observations](https://github.com/thaisontv123/jarvis/blob/master/app/memory/observations.py): `app/memory/observations.py`

### Other components

- **User**

## Connections

- User → Desktop shell: uses
- Desktop shell → Desktop UI: opens
- Desktop UI → Local API: calls same-origin
- Local API → Chat endpoint: routes requests (evidence: `app/main.py`)
- User → Chat endpoint: sends chat
- Chat endpoint → Agent orchestration: invokes (evidence: `app/api/routes_chat.py`)
- Voice session → Speech recognition: transcribes (evidence: `app/desktop/session.py`)
- Voice session → Agent orchestration: submits turns
- Agent orchestration → AI provider: requests responses
- Agent orchestration → Tool registry: dispatches tools (evidence: `app/core/orchestrator.py`)
- Tool registry → Permission engine: checks risk (evidence: `app/core/registry.py`)
- Agent orchestration → Confirmation: requests approval (evidence: `app/core/orchestrator.py`)
- Confirmation → Tool registry: executes approved (evidence: `app/api/routes_confirmations.py`)
- Confirmation → Action audit: records outcome (evidence: `app/api/routes_confirmations.py`)
- Chat endpoint → Application database: uses sessions (evidence: `app/api/routes_chat.py`)
- Tool registry → Capability packs: registers capabilities (evidence: `app/packs/builtin.py`)
- Capability packs → Computer control: includes computer (evidence: `app/packs/builtin.py`)
- Capability packs → External integrations: uses service clients (evidence: `app/integrations/runtime.py`)
- Agent orchestration → Memory retrieval: retrieves context
- Scheduler → Application database: reads and writes (evidence: `app/scheduler/engine.py`)
- Scheduler → Automated routines: runs schedules (evidence: `app/scheduler/runtime.py`)
- Scheduler → Speech output: delivers speech
- Observations → Application database: stores observations
- User → Voice session: speaks
