# eadmin2/jarvis_ai architecture

> Architecture diagram of the GitHub repository eadmin2/jarvis_ai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/eadmin2/jarvis_ai
- Repository: https://github.com/eadmin2/jarvis_ai

## Overview

J.A.R.V.I.S is a self-hosted voice assistant and browser HUD backed by Hermes Agent. The main workflow is microphone audio from the browser or push-to-talk client into the server, transcription, a persistent Hermes conversation, then streamed speech and HUD updates. The map also preserves documented approval, interruption, usage, dashboard, media-panel, and worker-monitoring capabilities. Server internals are represented as responsibilities of the same source file; unsampled implementation details are not inferred.

Main entry points: [`server/server.py`](https://github.com/eadmin2/jarvis_ai/blob/main/server/server.py) (Voice session pipeline), [`server/hud/index.html`](https://github.com/eadmin2/jarvis_ai/blob/main/server/hud/index.html) (Browser HUD), [`hermes-plugin/hud_display/__init__.py`](https://github.com/eadmin2/jarvis_ai/blob/main/hermes-plugin/hud_display/__init__.py) (HUD tool registration), [`client/client.py`](https://github.com/eadmin2/jarvis_ai/blob/main/client/client.py) (Push-to-talk client)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interaction["Voice &amp; HUD"]
  node_browser["Browser HUD<br/>[index.html]"]
  node_ptt["Push-to-talk client<br/>[client.py]"]
end

subgraph group_server["Voice Server"]
  node_voice["Voice session pipeline<br/>[server.py]"]
  node_stt["Speech transcription<br/>[server.py]"]
  node_agent_client["Hermes API client<br/>[server.py]"]
  node_safety["Auth &amp; approvals<br/>[server.py]"]
  node_speech["Streaming speech output<br/>[server.py]"]
  node_privacy["TTS privacy filter<br/>[server.py]"]
  node_usage[("Usage tracking<br/>[server.py]")]
  node_hud_api["HUD API &amp; events<br/>[server.py]"]
  node_dashboard["Dashboard proxy<br/>[server.py]"]
end

subgraph group_integrations["Agent &amp; Integrations"]
  node_plugin_registration["HUD tool registration<br/>[__init__.py]"]
  node_plugin_schemas["Display tool schemas<br/>[schemas.py]"]
  node_plugin_handlers["Display tool handlers<br/>[tools.py]"]
  node_hermes["Hermes Agent"]
  node_tts_provider["ElevenLabs TTS"]
  node_dashboard_service["Hermes dashboard"]
end

subgraph group_workers["Worker Services"]
  node_remote_stt["GPU transcription service<br/>[stt_server.py]"]
  node_worker_stats["Worker stats service<br/>[worker_stats.py]"]
end

node_human(("User"))

node_human -->|"speaks or types"| node_browser
node_browser -->|"streams audio"| node_voice
node_human -->|"speaks"| node_ptt
node_ptt -->|"streams audio"| node_voice
node_voice -->|"transcribes audio"| node_stt
node_stt -->|"tries remote STT"| node_remote_stt
node_voice -->|"dispatches turn"| node_agent_client
node_agent_client -->|"calls API"| node_hermes
node_voice -->|"gates requests"| node_safety
node_voice -->|"redacts TTS text"| node_privacy
node_voice -->|"streams reply audio"| node_speech
node_speech -->|"requests speech"| node_tts_provider
node_voice -->|"records usage"| node_usage
node_browser -->|"sends HUD actions"| node_hud_api
node_plugin_registration -->|"registers schemas"| node_plugin_schemas
node_plugin_registration -->|"registers handlers"| node_plugin_handlers
node_plugin_handlers -->|"posts panel actions"| node_hud_api
node_hermes -.->|"loads tools"| node_plugin_registration
node_hud_api -->|"proxies dashboards"| node_dashboard
node_dashboard -->|"proxies requests"| node_dashboard_service
node_voice -->|"fetches worker stats"| node_worker_stats

click node_browser "https://github.com/eadmin2/jarvis_ai/blob/main/server/hud/index.html"
click node_ptt "https://github.com/eadmin2/jarvis_ai/blob/main/client/client.py"
click node_voice "https://github.com/eadmin2/jarvis_ai/blob/main/server/server.py"
click node_stt "https://github.com/eadmin2/jarvis_ai/blob/main/server/server.py"
click node_agent_client "https://github.com/eadmin2/jarvis_ai/blob/main/server/server.py"
click node_safety "https://github.com/eadmin2/jarvis_ai/blob/main/server/server.py"
click node_speech "https://github.com/eadmin2/jarvis_ai/blob/main/server/server.py"
click node_privacy "https://github.com/eadmin2/jarvis_ai/blob/main/server/server.py"
click node_usage "https://github.com/eadmin2/jarvis_ai/blob/main/server/server.py"
click node_hud_api "https://github.com/eadmin2/jarvis_ai/blob/main/server/server.py"
click node_dashboard "https://github.com/eadmin2/jarvis_ai/blob/main/server/server.py"
click node_plugin_registration "https://github.com/eadmin2/jarvis_ai/blob/main/hermes-plugin/hud_display/__init__.py"
click node_plugin_schemas "https://github.com/eadmin2/jarvis_ai/blob/main/hermes-plugin/hud_display/schemas.py"
click node_plugin_handlers "https://github.com/eadmin2/jarvis_ai/blob/main/hermes-plugin/hud_display/tools.py"
click node_remote_stt "https://github.com/eadmin2/jarvis_ai/blob/main/worker/stt_server.py"
click node_worker_stats "https://github.com/eadmin2/jarvis_ai/blob/main/worker/worker_stats.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_browser,node_ptt,node_human toneBlue
class node_voice,node_stt,node_agent_client,node_safety,node_speech,node_privacy,node_usage,node_hud_api,node_dashboard toneAmber
class node_plugin_registration,node_plugin_schemas,node_plugin_handlers,node_hermes,node_tts_provider,node_dashboard_service toneMint
class node_remote_stt,node_worker_stats toneRose
```

## Components

### Voice & HUD

- [Browser HUD](https://github.com/eadmin2/jarvis_ai/blob/main/server/hud/index.html): `server/hud/index.html`
- [Push-to-talk client](https://github.com/eadmin2/jarvis_ai/blob/main/client/client.py): `client/client.py`

### Voice Server

- [Voice session pipeline](https://github.com/eadmin2/jarvis_ai/blob/main/server/server.py): `server/server.py`
- [Speech transcription](https://github.com/eadmin2/jarvis_ai/blob/main/server/server.py): `server/server.py`
- [Hermes API client](https://github.com/eadmin2/jarvis_ai/blob/main/server/server.py): `server/server.py`
- [Auth & approvals](https://github.com/eadmin2/jarvis_ai/blob/main/server/server.py): `server/server.py`
- [Streaming speech output](https://github.com/eadmin2/jarvis_ai/blob/main/server/server.py): `server/server.py`
- [TTS privacy filter](https://github.com/eadmin2/jarvis_ai/blob/main/server/server.py): `server/server.py`
- [Usage tracking](https://github.com/eadmin2/jarvis_ai/blob/main/server/server.py): `server/server.py`
- [HUD API & events](https://github.com/eadmin2/jarvis_ai/blob/main/server/server.py): `server/server.py`
- [Dashboard proxy](https://github.com/eadmin2/jarvis_ai/blob/main/server/server.py): `server/server.py`

### Agent & Integrations

- [HUD tool registration](https://github.com/eadmin2/jarvis_ai/blob/main/hermes-plugin/hud_display/__init__.py): `hermes-plugin/hud_display/__init__.py`
- [Display tool schemas](https://github.com/eadmin2/jarvis_ai/blob/main/hermes-plugin/hud_display/schemas.py): `hermes-plugin/hud_display/schemas.py`
- [Display tool handlers](https://github.com/eadmin2/jarvis_ai/blob/main/hermes-plugin/hud_display/tools.py): `hermes-plugin/hud_display/tools.py`
- **Hermes Agent**
- **ElevenLabs TTS**
- **Hermes dashboard**

### Worker Services

- [GPU transcription service](https://github.com/eadmin2/jarvis_ai/blob/main/worker/stt_server.py): `worker/stt_server.py`
- [Worker stats service](https://github.com/eadmin2/jarvis_ai/blob/main/worker/worker_stats.py): `worker/worker_stats.py`

### Other components

- **User**

## Connections

- User → Browser HUD: speaks or types (evidence: `README.md`)
- Browser HUD → Voice session pipeline: streams audio (evidence: `README.md`)
- User → Push-to-talk client: speaks (evidence: `README.md`)
- Push-to-talk client → Voice session pipeline: streams audio (evidence: `client/client.py`)
- Voice session pipeline → Speech transcription: transcribes audio (evidence: `server/server.py`)
- Speech transcription → GPU transcription service: tries remote STT (evidence: `server/server.py`)
- Voice session pipeline → Hermes API client: dispatches turn (evidence: `server/server.py`)
- Hermes API client → Hermes Agent: calls API (evidence: `README.md`)
- Voice session pipeline → Auth & approvals: gates requests (evidence: `server/server.py`)
- Voice session pipeline → TTS privacy filter: redacts TTS text (evidence: `README.md`)
- Voice session pipeline → Streaming speech output: streams reply audio (evidence: `README.md`)
- Streaming speech output → ElevenLabs TTS: requests speech (evidence: `README.md`)
- Voice session pipeline → Usage tracking: records usage (evidence: `server/server.py`)
- Browser HUD → HUD API & events: sends HUD actions (evidence: `README.md`)
- HUD tool registration → Display tool schemas: registers schemas (evidence: `hermes-plugin/hud_display/__init__.py`)
- HUD tool registration → Display tool handlers: registers handlers (evidence: `hermes-plugin/hud_display/__init__.py`)
- Display tool handlers → HUD API & events: posts panel actions (evidence: `hermes-plugin/hud_display/tools.py`)
- Hermes Agent → HUD tool registration: loads tools
- HUD API & events → Dashboard proxy: proxies dashboards (evidence: `README.md`)
- Dashboard proxy → Hermes dashboard: proxies requests (evidence: `server/server.py`)
- Voice session pipeline → Worker stats service: fetches worker stats (evidence: `README.md`)
