# harsh-life/jarvis architecture

> Architecture diagram of the GitHub repository harsh-life/jarvis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/harsh-life/jarvis
- Repository: https://github.com/harsh-life/jarvis

## Overview

JARVIS is a self-hosted assistant with an Android client and a server-side task runtime: users submit tasks, a model proposes actions, deterministic policy checks them, tools execute permitted steps, and risky actions wait for confirmation. The repository also includes device sign-in, configurable models, agent authoring, memory and vault, evaluation, reminders, voice, and a browser Workbench. The README documents the main workflow and optional capabilities; most implementation links are not verified by the sampled source, so uncertain connections are omitted.

Main entry points: [`server/agent/runtime.py`](https://github.com/harsh-life/jarvis/blob/main/server/agent/runtime.py) (Agent runtime), [`server/gateway/app.py`](https://github.com/harsh-life/jarvis/blob/main/server/gateway/app.py) (HTTP API), [`server/storage/models.py`](https://github.com/harsh-life/jarvis/blob/main/server/storage/models.py) (Persistent storage), [`android/app/src/main/kotlin/com/hypermind/jarvis/MainActivity.kt`](https://github.com/harsh-life/jarvis/blob/main/android/app/src/main/kotlin/com/hypermind/jarvis/MainActivity.kt) (Android app)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_client["Client surfaces"]
  node_android["Android app<br/>[MainActivity.kt]"]
  node_device_channel["Device channel<br/>[ChannelService.kt]"]
  node_device_perception["Screen perception"]
  node_workbench["Workbench<br/>[app.js]"]
  node_android_execution["Android actions"]
end

subgraph group_runtime["Task runtime"]
  node_task_runtime["Agent runtime<br/>[runtime.py]"]
  node_task_context["Run context<br/>[context.py]"]
  node_model_gateway["Model gateway<br/>[model_gateway.py]"]
  node_agent_service["Agent service<br/>[service.py]"]
  node_agent_compiler["Agent compiler<br/>[compiler.py]"]
  node_execution["Tool execution<br/>[executor.py]"]
  node_sandbox["Constrained execution<br/>[sandbox.py]"]
end

subgraph group_security["Identity and policy"]
  node_authorization["Authorization<br/>[authorization.py]"]
  node_capabilities["Capability policy<br/>[policy.py]"]
  node_auth["Device identity<br/>[sessions.py]"]
  node_graph_service["Graph lifecycle<br/>[service.py]"]
end

subgraph group_extensions["Optional capabilities"]
  node_memory["Persistent memory<br/>[provider.py]"]
  node_vault["Knowledge Vault<br/>[ingest.py]"]
  node_evaluation["Judge evaluation<br/>[service.py]"]
  node_reminders["Reminders<br/>[service.py]"]
  node_voice["Voice providers<br/>[service.py]"]
  node_push["Push wake<br/>[push.py]"]
end

subgraph group_platform["Server platform"]
  node_gateway["HTTP API<br/>[app.py]"]
  node_storage[("Persistent storage<br/>[models.py]")]
end

node_user(("User"))
node_model_provider["Model provider"]

node_user -->|"submits tasks"| node_android
node_user -->|"configures models"| node_workbench
node_android -.->|"sends requests"| node_gateway
node_workbench -->|"calls API"| node_gateway
node_gateway -.->|"authenticates"| node_auth
node_gateway -.->|"dispatches tasks"| node_task_runtime
node_task_runtime -.->|"uses context"| node_task_context
node_task_runtime -->|"requests proposals"| node_model_gateway
node_model_gateway -->|"queries model"| node_model_provider
node_task_runtime -->|"checks actions"| node_authorization
node_task_runtime -->|"executes tools"| node_execution
node_execution -->|"confines work"| node_sandbox
node_task_runtime -.->|"stores run state"| node_storage
node_task_runtime -.->|"uses agent definitions"| node_agent_service
node_agent_service -->|"compiles drafts"| node_agent_compiler
node_agent_service -.->|"persists definitions"| node_storage
node_authorization -.->|"checks policy"| node_capabilities
node_graph_service -.->|"updates memberships"| node_storage
node_graph_service -.->|"supports access checks"| node_authorization
node_android -->|"starts connection"| node_device_channel
node_device_channel -.->|"handles operations"| node_android_execution
node_device_channel -.->|"uses device state"| node_device_perception
node_task_runtime -.->|"uses memory"| node_memory
node_user -->|"curates knowledge"| node_vault
node_vault -.->|"indexes documents"| node_storage
node_task_runtime -.->|"evaluated by"| node_evaluation
node_user -->|"sets reminders"| node_reminders
node_user -.->|"speaks requests"| node_voice
node_user -.->|"receives wake"| node_push

click node_android "https://github.com/harsh-life/jarvis/blob/main/android/app/src/main/kotlin/com/hypermind/jarvis/MainActivity.kt"
click node_device_channel "https://github.com/harsh-life/jarvis/blob/main/android/app/src/main/kotlin/com/hypermind/jarvis/channel/ChannelService.kt"
click node_device_perception "https://github.com/harsh-life/jarvis/blob/main/android/app/src/main/kotlin/com/hypermind/jarvis/perception/JarvisAccessibilityService.kt"
click node_workbench "https://github.com/harsh-life/jarvis/blob/main/web/js/app.js"
click node_gateway "https://github.com/harsh-life/jarvis/blob/main/server/gateway/app.py"
click node_task_runtime "https://github.com/harsh-life/jarvis/blob/main/server/agent/runtime.py"
click node_task_context "https://github.com/harsh-life/jarvis/blob/main/server/agent/context.py"
click node_model_gateway "https://github.com/harsh-life/jarvis/blob/main/server/agents/gateway/model_gateway.py"
click node_agent_service "https://github.com/harsh-life/jarvis/blob/main/server/agents/service.py"
click node_agent_compiler "https://github.com/harsh-life/jarvis/blob/main/server/agents/compiler.py"
click node_authorization "https://github.com/harsh-life/jarvis/blob/main/server/graph/authorization.py"
click node_capabilities "https://github.com/harsh-life/jarvis/blob/main/server/capabilities/policy.py"
click node_auth "https://github.com/harsh-life/jarvis/blob/main/server/auth/sessions.py"
click node_execution "https://github.com/harsh-life/jarvis/blob/main/server/tools/executor.py"
click node_sandbox "https://github.com/harsh-life/jarvis/blob/main/server/fs/sandbox.py"
click node_storage "https://github.com/harsh-life/jarvis/blob/main/server/storage/models.py"
click node_graph_service "https://github.com/harsh-life/jarvis/blob/main/server/graph/service.py"
click node_android_execution "https://github.com/harsh-life/jarvis/blob/main/android/app/src/main/kotlin/com/hypermind/jarvis/channel/OperationHandler.kt"
click node_memory "https://github.com/harsh-life/jarvis/blob/main/server/memory/provider.py"
click node_vault "https://github.com/harsh-life/jarvis/blob/main/server/vault/ingest.py"
click node_evaluation "https://github.com/harsh-life/jarvis/blob/main/server/evaluation/service.py"
click node_reminders "https://github.com/harsh-life/jarvis/blob/main/server/scheduler/service.py"
click node_voice "https://github.com/harsh-life/jarvis/blob/main/server/voice/service.py"
click node_push "https://github.com/harsh-life/jarvis/blob/main/server/composition/push.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_android,node_device_channel,node_device_perception,node_workbench,node_android_execution,node_user toneBlue
class node_task_runtime,node_task_context,node_model_gateway,node_agent_service,node_agent_compiler,node_execution,node_sandbox toneAmber
class node_authorization,node_capabilities,node_auth,node_graph_service toneMint
class node_memory,node_vault,node_evaluation,node_reminders,node_voice,node_push toneRose
class node_gateway,node_storage,node_model_provider toneIndigo
```

## Components

### Client surfaces

- [Android app](https://github.com/harsh-life/jarvis/blob/main/android/app/src/main/kotlin/com/hypermind/jarvis/MainActivity.kt): `android/app/src/main/kotlin/com/hypermind/jarvis/MainActivity.kt`
- [Device channel](https://github.com/harsh-life/jarvis/blob/main/android/app/src/main/kotlin/com/hypermind/jarvis/channel/ChannelService.kt): `android/app/src/main/kotlin/com/hypermind/jarvis/channel/ChannelService.kt`
- [Screen perception](https://github.com/harsh-life/jarvis/blob/main/android/app/src/main/kotlin/com/hypermind/jarvis/perception/JarvisAccessibilityService.kt): `android/app/src/main/kotlin/com/hypermind/jarvis/perception/JarvisAccessibilityService.kt`
- [Workbench](https://github.com/harsh-life/jarvis/blob/main/web/js/app.js): `web/js/app.js`
- [Android actions](https://github.com/harsh-life/jarvis/blob/main/android/app/src/main/kotlin/com/hypermind/jarvis/channel/OperationHandler.kt): `android/app/src/main/kotlin/com/hypermind/jarvis/channel/OperationHandler.kt`

### Task runtime

- [Agent runtime](https://github.com/harsh-life/jarvis/blob/main/server/agent/runtime.py): `server/agent/runtime.py`
- [Run context](https://github.com/harsh-life/jarvis/blob/main/server/agent/context.py): `server/agent/context.py`
- [Model gateway](https://github.com/harsh-life/jarvis/blob/main/server/agents/gateway/model_gateway.py): `server/agents/gateway/model_gateway.py`
- [Agent service](https://github.com/harsh-life/jarvis/blob/main/server/agents/service.py): `server/agents/service.py`
- [Agent compiler](https://github.com/harsh-life/jarvis/blob/main/server/agents/compiler.py): `server/agents/compiler.py`
- [Tool execution](https://github.com/harsh-life/jarvis/blob/main/server/tools/executor.py): `server/tools/executor.py`
- [Constrained execution](https://github.com/harsh-life/jarvis/blob/main/server/fs/sandbox.py): `server/fs/sandbox.py`

### Identity and policy

- [Authorization](https://github.com/harsh-life/jarvis/blob/main/server/graph/authorization.py): `server/graph/authorization.py`
- [Capability policy](https://github.com/harsh-life/jarvis/blob/main/server/capabilities/policy.py): `server/capabilities/policy.py`
- [Device identity](https://github.com/harsh-life/jarvis/blob/main/server/auth/sessions.py): `server/auth/sessions.py`
- [Graph lifecycle](https://github.com/harsh-life/jarvis/blob/main/server/graph/service.py): `server/graph/service.py`

### Optional capabilities

- [Persistent memory](https://github.com/harsh-life/jarvis/blob/main/server/memory/provider.py): `server/memory/provider.py`
- [Knowledge Vault](https://github.com/harsh-life/jarvis/blob/main/server/vault/ingest.py): `server/vault/ingest.py`
- [Judge evaluation](https://github.com/harsh-life/jarvis/blob/main/server/evaluation/service.py): `server/evaluation/service.py`
- [Reminders](https://github.com/harsh-life/jarvis/blob/main/server/scheduler/service.py): `server/scheduler/service.py`
- [Voice providers](https://github.com/harsh-life/jarvis/blob/main/server/voice/service.py): `server/voice/service.py`
- [Push wake](https://github.com/harsh-life/jarvis/blob/main/server/composition/push.py): `server/composition/push.py`

### Server platform

- [HTTP API](https://github.com/harsh-life/jarvis/blob/main/server/gateway/app.py): `server/gateway/app.py`
- [Persistent storage](https://github.com/harsh-life/jarvis/blob/main/server/storage/models.py): `server/storage/models.py`

### Other components

- **User**
- **Model provider**

## Connections

- User → Android app: submits tasks (evidence: `README.md`)
- User → Workbench: configures models (evidence: `README.md`)
- Android app → HTTP API: sends requests
- Workbench → HTTP API: calls API (evidence: `web/js/app.js`)
- HTTP API → Device identity: authenticates
- HTTP API → Agent runtime: dispatches tasks
- Agent runtime → Run context: uses context
- Agent runtime → Model gateway: requests proposals (evidence: `README.md`)
- Model gateway → Model provider: queries model (evidence: `README.md`)
- Agent runtime → Authorization: checks actions (evidence: `README.md`)
- Agent runtime → Tool execution: executes tools (evidence: `README.md`)
- Tool execution → Constrained execution: confines work (evidence: `README.md`)
- Agent runtime → Persistent storage: stores run state
- Agent runtime → Agent service: uses agent definitions
- Agent service → Agent compiler: compiles drafts (evidence: `server/agents/service.py`)
- Agent service → Persistent storage: persists definitions (evidence: `server/agents/service.py`)
- Authorization → Capability policy: checks policy
- Graph lifecycle → Persistent storage: updates memberships (evidence: `server/graph/service.py`)
- Graph lifecycle → Authorization: supports access checks
- Android app → Device channel: starts connection (evidence: `android/app/src/main/kotlin/com/hypermind/jarvis/channel/ChannelService.kt`)
- Device channel → Android actions: handles operations
- Device channel → Screen perception: uses device state
- Agent runtime → Persistent memory: uses memory
- User → Knowledge Vault: curates knowledge (evidence: `server/vault/ingest.py`)
- Knowledge Vault → Persistent storage: indexes documents
- Agent runtime → Judge evaluation: evaluated by
- User → Reminders: sets reminders (evidence: `README.md`)
- User → Voice providers: speaks requests
- User → Push wake: receives wake
