# ruxakk/jarvis-voice-butler architecture

> Architecture diagram of the GitHub repository ruxakk/jarvis-voice-butler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ruxakk/jarvis-voice-butler
- Repository: https://github.com/ruxakk/jarvis-voice-butler

## Overview

Jarvis is a voice-operated AI butler: a user joins a LiveKit session, speaks with a realtime Gemini-backed agent, and can ask it to browse and act in a Playwright-controlled browser. The web client obtains LiveKit connection credentials and provides welcome, session, transcript, media-control, and visualizer interfaces; README also documents Flutter and console clients. Backend call edges are supported by the source index and excerpts; broader client-to-agent and provider relationships are documented in the README. Flutter implementation details are not present in the supplied tree.

Main entry points: [`jarvis_new/src/agent.py`](https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/src/agent.py) (Agent session server), [`jarvis_new/frontend`](https://github.com/ruxakk/jarvis-voice-butler/tree/main/jarvis_new/frontend) (Web client), [`jarvis_new/frontend/components/app/view-controller.tsx`](https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/frontend/components/app/view-controller.tsx) (View controller), [`jarvis_new/frontend/components/agents-ui/blocks/agent-session-view-01/components/agent-session-block.tsx`](https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/frontend/components/agents-ui/blocks/agent-session-view-01/components/agent-session-block.tsx) (Session interface)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_backend["Agent runtime"]
  node_agent_server["Agent session server<br/>[agent.py]"]
  node_persona["Butler instructions<br/>[prompts.py]"]
end

subgraph group_browser["Browser automation"]
  node_browser_tools["Browser function tools<br/>[tools.py]"]
  node_browser_manager["Browser manager<br/>[browser.py]"]
end

subgraph group_web["Web client"]
  node_web_client["Web client"]
  node_welcome["Welcome view<br/>[welcome-view.tsx]"]
  node_view_controller["View controller"]
  node_session_view["Session interface"]
  node_tile_view["Media tiles<br/>[tile-view.tsx]"]
  node_audio_visualizer["Audio visualizer"]
  node_visualizer_styles["Visualizer styles"]
  node_transcript_controls["Transcript and controls"]
  node_token_route["Token endpoint<br/>[route.ts]"]
end

subgraph group_mobile["Mobile client"]
  node_flutter_client["Flutter client"]
end

node_user(("User"))
node_livekit{{"LiveKit service"}}
node_realtime_model{{"Gemini realtime model"}}
node_chromium["Playwright Chromium"]
node_websites{{"Websites"}}
node_console_client["Console client<br/>[agent.py]"]

node_user -->|"uses"| node_web_client
node_web_client -.->|"requests credentials"| node_token_route
node_token_route -->|"issues access token"| node_livekit
node_web_client -->|"joins session"| node_livekit
node_web_client -->|"alternative client"| node_flutter_client
node_user -->|"uses"| node_flutter_client
node_user -->|"uses"| node_console_client
node_livekit -->|"dispatches session"| node_agent_server
node_agent_server -->|"connects session"| node_livekit
node_agent_server -->|"uses instructions"| node_persona
node_agent_server -->|"configures"| node_realtime_model
node_agent_server -->|"registers tools"| node_browser_tools
node_browser_tools -->|"invokes browser actions"| node_browser_manager
node_browser_manager -->|"controls"| node_chromium
node_chromium -->|"browses"| node_websites
node_web_client -.->|"renders views"| node_view_controller
node_view_controller -->|"shows before connection"| node_welcome
node_view_controller -->|"shows when connected"| node_session_view
node_session_view -->|"renders media tiles"| node_tile_view
node_session_view -->|"renders transcript"| node_transcript_controls
node_tile_view -->|"renders visualizer"| node_audio_visualizer
node_audio_visualizer -->|"selects style"| node_visualizer_styles

click node_web_client "https://github.com/ruxakk/jarvis-voice-butler/tree/main/jarvis_new/frontend"
click node_welcome "https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/frontend/components/app/welcome-view.tsx"
click node_view_controller "https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/frontend/components/app/view-controller.tsx"
click node_session_view "https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/frontend/components/agents-ui/blocks/agent-session-view-01/components/agent-session-block.tsx"
click node_tile_view "https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/frontend/components/agents-ui/blocks/agent-session-view-01/components/tile-view.tsx"
click node_audio_visualizer "https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/frontend/components/agents-ui/blocks/agent-session-view-01/components/audio-visualizer.tsx"
click node_visualizer_styles "https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/frontend/components/agents-ui/agent-audio-visualizer-aura.tsx"
click node_transcript_controls "https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/frontend/components/agents-ui/agent-chat-transcript.tsx"
click node_token_route "https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/frontend/app/api/token/route.ts"
click node_agent_server "https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/src/agent.py"
click node_persona "https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/src/prompts.py"
click node_browser_tools "https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/src/tools.py"
click node_browser_manager "https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/src/browser.py"
click node_flutter_client "https://github.com/ruxakk/jarvis-voice-butler/tree/main/agent-starter-flutter"
click node_console_client "https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/src/agent.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_agent_server,node_persona,node_user,node_console_client toneBlue
class node_browser_tools,node_browser_manager toneAmber
class node_web_client,node_welcome,node_view_controller,node_session_view,node_tile_view,node_audio_visualizer,node_visualizer_styles,node_transcript_controls,node_token_route toneMint
class node_flutter_client toneRose
class node_livekit,node_realtime_model,node_chromium,node_websites toneIndigo
```

## Components

### Agent runtime

- [Agent session server](https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/src/agent.py): `jarvis_new/src/agent.py`
- [Butler instructions](https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/src/prompts.py): `jarvis_new/src/prompts.py`

### Browser automation

- [Browser function tools](https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/src/tools.py): `jarvis_new/src/tools.py`
- [Browser manager](https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/src/browser.py): `jarvis_new/src/browser.py`

### Web client

- [Web client](https://github.com/ruxakk/jarvis-voice-butler/tree/main/jarvis_new/frontend): `jarvis_new/frontend`
- [Welcome view](https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/frontend/components/app/welcome-view.tsx): `jarvis_new/frontend/components/app/welcome-view.tsx`
- [View controller](https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/frontend/components/app/view-controller.tsx): `jarvis_new/frontend/components/app/view-controller.tsx`
- [Session interface](https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/frontend/components/agents-ui/blocks/agent-session-view-01/components/agent-session-block.tsx): `jarvis_new/frontend/components/agents-ui/blocks/agent-session-view-01/components/agent-session-block.tsx`
- [Media tiles](https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/frontend/components/agents-ui/blocks/agent-session-view-01/components/tile-view.tsx): `jarvis_new/frontend/components/agents-ui/blocks/agent-session-view-01/components/tile-view.tsx`
- [Audio visualizer](https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/frontend/components/agents-ui/blocks/agent-session-view-01/components/audio-visualizer.tsx): `jarvis_new/frontend/components/agents-ui/blocks/agent-session-view-01/components/audio-visualizer.tsx`
- [Visualizer styles](https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/frontend/components/agents-ui/agent-audio-visualizer-aura.tsx): `jarvis_new/frontend/components/agents-ui/agent-audio-visualizer-aura.tsx`
- [Transcript and controls](https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/frontend/components/agents-ui/agent-chat-transcript.tsx): `jarvis_new/frontend/components/agents-ui/agent-chat-transcript.tsx`
- [Token endpoint](https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/frontend/app/api/token/route.ts): `jarvis_new/frontend/app/api/token/route.ts`

### Mobile client

- [Flutter client](https://github.com/ruxakk/jarvis-voice-butler/tree/main/agent-starter-flutter): `agent-starter-flutter`

### Other components

- **User**
- **LiveKit service**
- **Gemini realtime model**
- **Playwright Chromium**
- **Websites**
- [Console client](https://github.com/ruxakk/jarvis-voice-butler/blob/main/jarvis_new/src/agent.py): `jarvis_new/src/agent.py`

## Connections

- User → Web client: uses
- Web client → Token endpoint: requests credentials
- Token endpoint → LiveKit service: issues access token (evidence: `jarvis_new/frontend/app/api/token/route.ts`)
- Web client → LiveKit service: joins session
- Web client → Flutter client: alternative client
- User → Flutter client: uses
- User → Console client: uses
- LiveKit service → Agent session server: dispatches session
- Agent session server → LiveKit service: connects session (evidence: `jarvis_new/src/agent.py`)
- Agent session server → Butler instructions: uses instructions (evidence: `jarvis_new/src/agent.py`)
- Agent session server → Gemini realtime model: configures (evidence: `jarvis_new/src/agent.py`)
- Agent session server → Browser function tools: registers tools (evidence: `jarvis_new/src/agent.py`)
- Browser function tools → Browser manager: invokes browser actions (evidence: `jarvis_new/src/tools.py`)
- Browser manager → Playwright Chromium: controls (evidence: `jarvis_new/src/browser.py`)
- Playwright Chromium → Websites: browses
- Web client → View controller: renders views
- View controller → Welcome view: shows before connection (evidence: `jarvis_new/frontend/components/app/view-controller.tsx`)
- View controller → Session interface: shows when connected (evidence: `jarvis_new/frontend/components/app/view-controller.tsx`)
- Session interface → Media tiles: renders media tiles (evidence: `jarvis_new/frontend/components/agents-ui/blocks/agent-session-view-01/components/agent-session-block.tsx`)
- Session interface → Transcript and controls: renders transcript (evidence: `jarvis_new/frontend/components/agents-ui/blocks/agent-session-view-01/components/agent-session-block.tsx`)
- Media tiles → Audio visualizer: renders visualizer (evidence: `jarvis_new/frontend/components/agents-ui/blocks/agent-session-view-01/components/tile-view.tsx`)
- Audio visualizer → Visualizer styles: selects style (evidence: `jarvis_new/frontend/components/agents-ui/blocks/agent-session-view-01/components/audio-visualizer.tsx`)
