# arpy8/esp32_voice_assistant architecture

> Architecture diagram of the GitHub repository arpy8/esp32_voice_assistant, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arpy8/esp32_voice_assistant
- Repository: https://github.com/arpy8/esp32_voice_assistant

## Overview

This repository implements an ESP32 voice assistant: a button-triggered recording is streamed to a Python WebSocket server, processed into a spoken response, and played back on the device. The backend excerpt shows the request handler, processing pipeline, model calls, and audio streaming; the README documents the device workflow and major hardware and AI capabilities. The firmware source was not excerpted, so its internal call structure is not shown.

Main entry points: [`server/main.py`](https://github.com/arpy8/esp32_voice_assistant/blob/main/server/main.py) (Audio Pipeline), [`esp-code/esp-code.ino`](https://github.com/arpy8/esp32_voice_assistant/blob/main/esp-code/esp-code.ino) (ESP32 Firmware)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_device["ESP32 Device"]
  node_mic["INMP441 Mic"]
  node_firmware["ESP32 Firmware<br/>[esp-code.ino]"]
  node_speaker["Speaker Output"]
end

subgraph group_transport["Audio Transport"]
  node_ws["WebSocket Link"]
  node_stream["Audio Streaming<br/>[main.py]"]
end

subgraph group_backend["Assistant Backend"]
  node_handler["Client Handler<br/>[main.py]"]
  node_audio_buffer["Audio Buffer<br/>[main.py]"]
  node_pipeline["Audio Pipeline<br/>[main.py]"]
  node_history[("Conversation History<br/>[main.py]")]
end

subgraph group_models["AI Models"]
  node_stt["Speech Recognition<br/>[main.py]"]
  node_llm["Reply Generation<br/>[main.py]"]
  node_tts["Speech Synthesis<br/>[main.py]"]
end

node_user(("User Speech"))
node_gemini{{"Google Gemini"}}
node_piper{{"Piper TTS"}}

node_user -->|"speaks to"| node_mic
node_mic -->|"audio input"| node_firmware
node_firmware -->|"streams audio"| node_ws
node_ws -->|"delivers audio"| node_handler
node_handler -->|"buffers audio"| node_audio_buffer
node_handler -->|"invokes processing"| node_pipeline
node_pipeline -->|"transcribes audio"| node_stt
node_pipeline -->|"requests reply"| node_llm
node_llm -->|"uses context"| node_history
node_llm -->|"generates response"| node_gemini
node_pipeline -->|"synthesizes reply"| node_tts
node_tts -->|"uses voice model"| node_piper
node_pipeline -->|"streams generated audio"| node_stream
node_stream -->|"sends audio chunks"| node_ws
node_ws -->|"returns audio"| node_firmware
node_firmware -->|"plays audio"| node_speaker

click node_firmware "https://github.com/arpy8/esp32_voice_assistant/blob/main/esp-code/esp-code.ino"
click node_handler "https://github.com/arpy8/esp32_voice_assistant/blob/main/server/main.py"
click node_audio_buffer "https://github.com/arpy8/esp32_voice_assistant/blob/main/server/main.py"
click node_pipeline "https://github.com/arpy8/esp32_voice_assistant/blob/main/server/main.py"
click node_stt "https://github.com/arpy8/esp32_voice_assistant/blob/main/server/main.py"
click node_llm "https://github.com/arpy8/esp32_voice_assistant/blob/main/server/main.py"
click node_history "https://github.com/arpy8/esp32_voice_assistant/blob/main/server/main.py"
click node_tts "https://github.com/arpy8/esp32_voice_assistant/blob/main/server/main.py"
click node_stream "https://github.com/arpy8/esp32_voice_assistant/blob/main/server/main.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_mic,node_firmware,node_speaker,node_user toneBlue
class node_ws,node_stream toneAmber
class node_handler,node_audio_buffer,node_pipeline,node_history toneMint
class node_stt,node_llm,node_tts toneRose
class node_gemini,node_piper toneIndigo
```

## Components

### ESP32 Device

- **INMP441 Mic**
- [ESP32 Firmware](https://github.com/arpy8/esp32_voice_assistant/blob/main/esp-code/esp-code.ino): `esp-code/esp-code.ino`
- **Speaker Output**

### Audio Transport

- **WebSocket Link**
- [Audio Streaming](https://github.com/arpy8/esp32_voice_assistant/blob/main/server/main.py): `server/main.py`

### Assistant Backend

- [Client Handler](https://github.com/arpy8/esp32_voice_assistant/blob/main/server/main.py): `server/main.py`
- [Audio Buffer](https://github.com/arpy8/esp32_voice_assistant/blob/main/server/main.py): `server/main.py`
- [Audio Pipeline](https://github.com/arpy8/esp32_voice_assistant/blob/main/server/main.py): `server/main.py`
- [Conversation History](https://github.com/arpy8/esp32_voice_assistant/blob/main/server/main.py): `server/main.py`

### AI Models

- [Speech Recognition](https://github.com/arpy8/esp32_voice_assistant/blob/main/server/main.py): `server/main.py`
- [Reply Generation](https://github.com/arpy8/esp32_voice_assistant/blob/main/server/main.py): `server/main.py`
- [Speech Synthesis](https://github.com/arpy8/esp32_voice_assistant/blob/main/server/main.py): `server/main.py`

### Other components

- **User Speech**
- **Google Gemini**
- **Piper TTS**

## Connections

- User Speech → INMP441 Mic: speaks to (evidence: `README.md`)
- INMP441 Mic → ESP32 Firmware: audio input (evidence: `README.md`)
- ESP32 Firmware → WebSocket Link: streams audio (evidence: `README.md`)
- WebSocket Link → Client Handler: delivers audio (evidence: `README.md`)
- Client Handler → Audio Buffer: buffers audio (evidence: `server/main.py`)
- Client Handler → Audio Pipeline: invokes processing (evidence: `server/main.py`)
- Audio Pipeline → Speech Recognition: transcribes audio (evidence: `server/main.py`)
- Audio Pipeline → Reply Generation: requests reply (evidence: `server/main.py`)
- Reply Generation → Conversation History: uses context (evidence: `server/main.py`)
- Reply Generation → Google Gemini: generates response (evidence: `server/main.py`)
- Audio Pipeline → Speech Synthesis: synthesizes reply (evidence: `server/main.py`)
- Speech Synthesis → Piper TTS: uses voice model (evidence: `server/main.py`)
- Audio Pipeline → Audio Streaming: streams generated audio (evidence: `server/main.py`)
- Audio Streaming → WebSocket Link: sends audio chunks (evidence: `server/main.py`)
- WebSocket Link → ESP32 Firmware: returns audio (evidence: `README.md`)
- ESP32 Firmware → Speaker Output: plays audio (evidence: `README.md`)
