# ownding/open-xiaoai architecture

> Architecture diagram of the GitHub repository ownding/open-xiaoai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ownding/open-xiaoai
- Repository: https://github.com/ownding/open-xiaoai

## Overview

Open-XiaoAI modifies Xiaomi smart speakers so their microphone/audio and device controls can participate in AI voice workflows. The principal path is a speaker running the Rust client, which exchanges messages with a voice service; optional integrations add Hermes reasoning and Home Assistant control, while Sherpa provides an alternate TTS deployment. The repository also includes firmware-patch tooling and a separate promo-video production project. Internal edges are limited to relationships shown in excerpts or the source index; unsampled README-described capabilities are represented without inferred wiring.

Main entry points: [`packages/client-rust/src/services/connect/message.rs`](https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/services/connect/message.rs) (WebSocket Transport), [`packages/client-rust/src/services/connect/handler.rs`](https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/services/connect/handler.rs) (Message Dispatch), [`deploy/hermes/xiaozhi-provider/hermes.py`](https://github.com/ownding/open-xiaoai/blob/main/deploy/hermes/xiaozhi-provider/hermes.py) (XiaoZhi Service), [`packages/client-rust/src/services/connect/rpc.rs`](https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/services/connect/rpc.rs) (RPC)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_device["Speaker Client"]
  node_client_entry["Client Runtime<br/>[client.rs]"]
  node_audio_capture["Audio Capture<br/>[record.rs]"]
  node_audio_playback["Audio Playback<br/>[play.rs]"]
  node_audio_config["Audio Config<br/>[config.rs]"]
  node_ws_transport["WebSocket Transport<br/>[message.rs]"]
  node_message_dispatch["Message Dispatch<br/>[handler.rs]"]
  node_rpc_layer["RPC<br/>[rpc.rs]"]
  node_message_types["Message Types<br/>[data.rs]"]
  node_speaker_control["Speaker Control<br/>[speaker.rs]"]
  node_shell_utils["Shell Utilities<br/>[shell.rs]"]
  node_patch_ota["OTA Patch Tool<br/>[ota.ts]"]
end

subgraph group_voice["Voice Integrations"]
  node_xiaozhi["XiaoZhi Service<br/>[hermes.py]"]
  node_hermes["Hermes Provider<br/>[hermes.py]"]
  node_tts_server["Sherpa TTS<br/>[server.py]"]
  node_tts_modes["TTS Mode Switching<br/>[switch_config.py]"]
  node_llm_switch["LLM Switching<br/>[switch_llm.py]"]
end

subgraph group_home["Home Automation"]
  node_ha["Home Assistant"]
  node_ha_onboard["HA Onboarding<br/>[onboard.py]"]
  node_ha_devices["Device Mapping<br/>[ha_device_table.py]"]
end

subgraph group_promo["Promo Production"]
  node_video_timeline["Video Timeline<br/>[timeline.py]"]
  node_video_scene["3D Scene<br/>[scene.py]"]
  node_video_plates["Shot Plates<br/>[plates.py]"]
  node_video_render["Plate Renderer<br/>[render_plate.py]"]
  node_audio_synth["Audio Synth<br/>[synth.py]"]
  node_audio_score["Music Score<br/>[score.py]"]
  node_audio_mix["Audio Mix<br/>[mix.py]"]
  node_video_compositor["Video Compositor<br/>[comp.js]"]
  node_video_render_entry["Render Entry<br/>[render.mjs]"]
end

node_speaker_user(("Speaker User"))
node_xiaomi_speaker(("Xiaomi Speaker"))

node_speaker_user -->|"speaks to"| node_xiaomi_speaker
node_xiaomi_speaker -->|"runs client"| node_client_entry
node_client_entry -.->|"uses transport"| node_ws_transport
node_audio_capture -->|"reads config"| node_audio_config
node_audio_playback -->|"reads config"| node_audio_config
node_ws_transport -->|"serializes messages"| node_message_types
node_ws_transport -->|"dispatches messages"| node_message_dispatch
node_ws_transport -->|"sends requests"| node_rpc_layer
node_message_dispatch -->|"routes requests"| node_rpc_layer
node_message_dispatch -->|"sends responses"| node_ws_transport
node_speaker_control -->|"calls remote"| node_rpc_layer
node_speaker_control -->|"uses commands"| node_shell_utils
node_xiaomi_speaker -->|"streams voice"| node_xiaozhi
node_xiaozhi -->|"delegates reasoning"| node_hermes
node_hermes -->|"controls devices"| node_ha
node_ha_onboard -.->|"onboards"| node_ha
node_ha_devices -.->|"maps devices"| node_ha
node_xiaozhi -.->|"uses alternate TTS"| node_tts_server
node_tts_modes -.->|"switches mode"| node_tts_server
node_llm_switch -.->|"selects provider"| node_hermes
node_patch_ota -.->|"updates firmware"| node_xiaomi_speaker
node_video_render -->|"renders plates"| node_video_plates
node_video_render -->|"uses scene"| node_video_scene
node_video_plates -->|"stages shots"| node_video_scene
node_audio_score -->|"uses synthesis"| node_audio_synth
node_audio_mix -->|"uses synthesis"| node_audio_synth
node_audio_mix -->|"mixes score"| node_audio_score

click node_client_entry "https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/bin/client.rs"
click node_audio_capture "https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/services/audio/record.rs"
click node_audio_playback "https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/services/audio/play.rs"
click node_audio_config "https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/services/audio/config.rs"
click node_ws_transport "https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/services/connect/message.rs"
click node_message_dispatch "https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/services/connect/handler.rs"
click node_rpc_layer "https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/services/connect/rpc.rs"
click node_message_types "https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/services/connect/data.rs"
click node_speaker_control "https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/services/speaker.rs"
click node_shell_utils "https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/utils/shell.rs"
click node_xiaozhi "https://github.com/ownding/open-xiaoai/blob/main/deploy/hermes/xiaozhi-provider/hermes.py"
click node_hermes "https://github.com/ownding/open-xiaoai/blob/main/deploy/hermes/xiaozhi-provider/hermes.py"
click node_tts_server "https://github.com/ownding/open-xiaoai/blob/main/deploy/sherpa-tts/server.py"
click node_tts_modes "https://github.com/ownding/open-xiaoai/blob/main/deploy/sherpa-tts/switch_config.py"
click node_ha_onboard "https://github.com/ownding/open-xiaoai/blob/main/deploy/hermes/demo-ha/onboard.py"
click node_ha_devices "https://github.com/ownding/open-xiaoai/blob/main/deploy/hermes/ha_device_table.py"
click node_llm_switch "https://github.com/ownding/open-xiaoai/blob/main/deploy/hermes/switch_llm.py"
click node_patch_ota "https://github.com/ownding/open-xiaoai/blob/main/packages/client-patch/src/ota.ts"
click node_video_timeline "https://github.com/ownding/open-xiaoai/blob/main/promo-video/src/timeline.py"
click node_video_scene "https://github.com/ownding/open-xiaoai/blob/main/promo-video/src/blender/scene.py"
click node_video_plates "https://github.com/ownding/open-xiaoai/blob/main/promo-video/src/blender/plates.py"
click node_video_render "https://github.com/ownding/open-xiaoai/blob/main/promo-video/src/blender/render_plate.py"
click node_audio_synth "https://github.com/ownding/open-xiaoai/blob/main/promo-video/src/audio/synth.py"
click node_audio_score "https://github.com/ownding/open-xiaoai/blob/main/promo-video/src/audio/score.py"
click node_audio_mix "https://github.com/ownding/open-xiaoai/blob/main/promo-video/src/audio/mix.py"
click node_video_compositor "https://github.com/ownding/open-xiaoai/blob/main/promo-video/src/compositor/comp.js"
click node_video_render_entry "https://github.com/ownding/open-xiaoai/blob/main/promo-video/src/compositor/render.mjs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_client_entry,node_audio_capture,node_audio_playback,node_audio_config,node_ws_transport,node_message_dispatch,node_rpc_layer,node_message_types,node_speaker_control,node_shell_utils,node_patch_ota,node_speaker_user toneBlue
class node_xiaozhi,node_hermes,node_tts_server,node_tts_modes,node_llm_switch toneAmber
class node_ha,node_ha_onboard,node_ha_devices toneMint
class node_video_timeline,node_video_scene,node_video_plates,node_video_render,node_audio_synth,node_audio_score,node_audio_mix,node_video_compositor,node_video_render_entry toneRose
class node_xiaomi_speaker toneIndigo
```

## Components

### Speaker Client

- [Client Runtime](https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/bin/client.rs): `packages/client-rust/src/bin/client.rs`
- [Audio Capture](https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/services/audio/record.rs): `packages/client-rust/src/services/audio/record.rs`
- [Audio Playback](https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/services/audio/play.rs): `packages/client-rust/src/services/audio/play.rs`
- [Audio Config](https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/services/audio/config.rs): `packages/client-rust/src/services/audio/config.rs`
- [WebSocket Transport](https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/services/connect/message.rs): `packages/client-rust/src/services/connect/message.rs`
- [Message Dispatch](https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/services/connect/handler.rs): `packages/client-rust/src/services/connect/handler.rs`
- [RPC](https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/services/connect/rpc.rs): `packages/client-rust/src/services/connect/rpc.rs`
- [Message Types](https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/services/connect/data.rs): `packages/client-rust/src/services/connect/data.rs`
- [Speaker Control](https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/services/speaker.rs): `packages/client-rust/src/services/speaker.rs`
- [Shell Utilities](https://github.com/ownding/open-xiaoai/blob/main/packages/client-rust/src/utils/shell.rs): `packages/client-rust/src/utils/shell.rs`
- [OTA Patch Tool](https://github.com/ownding/open-xiaoai/blob/main/packages/client-patch/src/ota.ts): `packages/client-patch/src/ota.ts`

### Voice Integrations

- [XiaoZhi Service](https://github.com/ownding/open-xiaoai/blob/main/deploy/hermes/xiaozhi-provider/hermes.py): `deploy/hermes/xiaozhi-provider/hermes.py`
- [Hermes Provider](https://github.com/ownding/open-xiaoai/blob/main/deploy/hermes/xiaozhi-provider/hermes.py): `deploy/hermes/xiaozhi-provider/hermes.py`
- [Sherpa TTS](https://github.com/ownding/open-xiaoai/blob/main/deploy/sherpa-tts/server.py): `deploy/sherpa-tts/server.py`
- [TTS Mode Switching](https://github.com/ownding/open-xiaoai/blob/main/deploy/sherpa-tts/switch_config.py): `deploy/sherpa-tts/switch_config.py`
- [LLM Switching](https://github.com/ownding/open-xiaoai/blob/main/deploy/hermes/switch_llm.py): `deploy/hermes/switch_llm.py`

### Home Automation

- **Home Assistant**
- [HA Onboarding](https://github.com/ownding/open-xiaoai/blob/main/deploy/hermes/demo-ha/onboard.py): `deploy/hermes/demo-ha/onboard.py`
- [Device Mapping](https://github.com/ownding/open-xiaoai/blob/main/deploy/hermes/ha_device_table.py): `deploy/hermes/ha_device_table.py`

### Promo Production

- [Video Timeline](https://github.com/ownding/open-xiaoai/blob/main/promo-video/src/timeline.py): `promo-video/src/timeline.py`
- [3D Scene](https://github.com/ownding/open-xiaoai/blob/main/promo-video/src/blender/scene.py): `promo-video/src/blender/scene.py`
- [Shot Plates](https://github.com/ownding/open-xiaoai/blob/main/promo-video/src/blender/plates.py): `promo-video/src/blender/plates.py`
- [Plate Renderer](https://github.com/ownding/open-xiaoai/blob/main/promo-video/src/blender/render_plate.py): `promo-video/src/blender/render_plate.py`
- [Audio Synth](https://github.com/ownding/open-xiaoai/blob/main/promo-video/src/audio/synth.py): `promo-video/src/audio/synth.py`
- [Music Score](https://github.com/ownding/open-xiaoai/blob/main/promo-video/src/audio/score.py): `promo-video/src/audio/score.py`
- [Audio Mix](https://github.com/ownding/open-xiaoai/blob/main/promo-video/src/audio/mix.py): `promo-video/src/audio/mix.py`
- [Video Compositor](https://github.com/ownding/open-xiaoai/blob/main/promo-video/src/compositor/comp.js): `promo-video/src/compositor/comp.js`
- [Render Entry](https://github.com/ownding/open-xiaoai/blob/main/promo-video/src/compositor/render.mjs): `promo-video/src/compositor/render.mjs`

### Other components

- **Speaker User**
- **Xiaomi Speaker**

## Connections

- Speaker User → Xiaomi Speaker: speaks to (evidence: `README.md`)
- Xiaomi Speaker → Client Runtime: runs client (evidence: `README.md`)
- Client Runtime → WebSocket Transport: uses transport
- Audio Capture → Audio Config: reads config (evidence: `packages/client-rust/src/services/audio/record.rs`)
- Audio Playback → Audio Config: reads config (evidence: `packages/client-rust/src/services/audio/play.rs`)
- WebSocket Transport → Message Types: serializes messages (evidence: `packages/client-rust/src/services/connect/message.rs`)
- WebSocket Transport → Message Dispatch: dispatches messages (evidence: `packages/client-rust/src/services/connect/message.rs`)
- WebSocket Transport → RPC: sends requests (evidence: `packages/client-rust/src/services/connect/message.rs`)
- Message Dispatch → RPC: routes requests (evidence: `packages/client-rust/src/services/connect/handler.rs`)
- Message Dispatch → WebSocket Transport: sends responses (evidence: `packages/client-rust/src/services/connect/handler.rs`)
- Speaker Control → RPC: calls remote (evidence: `packages/client-rust/src/services/speaker.rs`)
- Speaker Control → Shell Utilities: uses commands (evidence: `packages/client-rust/src/services/speaker.rs`)
- Xiaomi Speaker → XiaoZhi Service: streams voice (evidence: `README.md`)
- XiaoZhi Service → Hermes Provider: delegates reasoning (evidence: `README.md`)
- Hermes Provider → Home Assistant: controls devices (evidence: `README.md`)
- HA Onboarding → Home Assistant: onboards
- Device Mapping → Home Assistant: maps devices
- XiaoZhi Service → Sherpa TTS: uses alternate TTS
- TTS Mode Switching → Sherpa TTS: switches mode
- LLM Switching → Hermes Provider: selects provider
- OTA Patch Tool → Xiaomi Speaker: updates firmware
- Plate Renderer → Shot Plates: renders plates (evidence: `promo-video/src/blender/render_plate.py`)
- Plate Renderer → 3D Scene: uses scene (evidence: `promo-video/src/blender/render_plate.py`)
- Shot Plates → 3D Scene: stages shots (evidence: `promo-video/src/blender/plates.py`)
- Music Score → Audio Synth: uses synthesis (evidence: `promo-video/src/audio/score.py`)
- Audio Mix → Audio Synth: uses synthesis (evidence: `promo-video/src/audio/mix.py`)
- Audio Mix → Music Score: mixes score (evidence: `promo-video/src/audio/mix.py`)
