How ruxakk/jarvis-voice-butler works
Jarvis is a voice-operated AI butler: a user joins a LiveKit session, speaks with a realtime Gemini-backed agent, and can ask it to browse and act in a Playwright-controlled browser. The web client obtains LiveKit connection credentials and provides welcome, session, transcript, media-control, and visualizer interfaces; README also documents Flutter and console clients. Backend call edges are supported by the source index and excerpts; broader client-to-agent and provider relationships are documented in the README. Flutter implementation details are not present in the supplied tree.
Main entry points: agent.py Agent session server · frontend Web client · view-controller.tsx View controller · agent-session-block.tsx Session interface
Components 20
Agent runtime
- Agent session server
jarvis_new/src/agent.py - Butler instructions
jarvis_new/src/prompts.py
Browser automation
- Browser function tools
jarvis_new/src/tools.py - Browser manager
jarvis_new/src/browser.py
Web client
- Web client
jarvis_new/frontend - Welcome view
jarvis_new/frontend/components/app/welcome-view.tsx - View controller
jarvis_new/frontend/components/app/view-controller.tsx - Session interface
jarvis_new/frontend/components/agents-ui/blocks/agent-session-view-01/components/agent-session-block.tsx - Media tiles
jarvis_new/frontend/components/agents-ui/blocks/agent-session-view-01/components/tile-view.tsx - Audio visualizer
jarvis_new/frontend/components/agents-ui/blocks/agent-session-view-01/components/audio-visualizer.tsx - Visualizer styles
jarvis_new/frontend/components/agents-ui/agent-audio-visualizer-aura.tsx - Transcript and controls
jarvis_new/frontend/components/agents-ui/agent-chat-transcript.tsx - Token endpoint
jarvis_new/frontend/app/api/token/route.ts
Mobile client
- Flutter client
agent-starter-flutter
Other components
- User
- LiveKit service
- Gemini realtime model
- Playwright Chromium
- Websites
- Console client
jarvis_new/src/agent.py
Connections 22
- User → Web client: uses
- Web client → Token endpoint: requests credentials
- Token endpoint → LiveKit service: issues access token
jarvis_new/frontend/app/api/token/route.ts - Web client → LiveKit service: joins session
- Web client → Flutter client: alternative client
- User → Flutter client: uses
- User → Console client: uses
- LiveKit service → Agent session server: dispatches session
- Agent session server → LiveKit service: connects session
jarvis_new/src/agent.py - Agent session server → Butler instructions: uses instructions
jarvis_new/src/agent.py - Agent session server → Gemini realtime model: configures
jarvis_new/src/agent.py - Agent session server → Browser function tools: registers tools
jarvis_new/src/agent.py - Browser function tools → Browser manager: invokes browser actions
jarvis_new/src/tools.py - Browser manager → Playwright Chromium: controls
jarvis_new/src/browser.py - Playwright Chromium → Websites: browses
- Web client → View controller: renders views
- View controller → Welcome view: shows before connection
jarvis_new/frontend/components/app/view-controller.tsx - View controller → Session interface: shows when connected
jarvis_new/frontend/components/app/view-controller.tsx - Session interface → Media tiles: renders media tiles
jarvis_new/frontend/components/agents-ui/blocks/agent-session-view-01/components/agent-session-block.tsx - Session interface → Transcript and controls: renders transcript
jarvis_new/frontend/components/agents-ui/blocks/agent-session-view-01/components/agent-session-block.tsx - Media tiles → Audio visualizer: renders visualizer
jarvis_new/frontend/components/agents-ui/blocks/agent-session-view-01/components/tile-view.tsx - Audio visualizer → Visualizer styles: selects style
jarvis_new/frontend/components/agents-ui/blocks/agent-session-view-01/components/audio-visualizer.tsx