How arpy8/esp32_voice_assistant works
This repository implements an ESP32 voice assistant: a button-triggered recording is streamed to a Python WebSocket server, processed into a spoken response, and played back on the device. The backend excerpt shows the request handler, processing pipeline, model calls, and audio streaming; the README documents the device workflow and major hardware and AI capabilities. The firmware source was not excerpted, so its internal call structure is not shown.
Main entry points: main.py Audio Pipeline · esp-code.ino ESP32 Firmware
Components 15
ESP32 Device
- INMP441 Mic
- ESP32 Firmware
esp-code/esp-code.ino - Speaker Output
Audio Transport
- WebSocket Link
- Audio Streaming
server/main.py
Assistant Backend
- Client Handler
server/main.py - Audio Buffer
server/main.py - Audio Pipeline
server/main.py - Conversation History
server/main.py
AI Models
- Speech Recognition
server/main.py - Reply Generation
server/main.py - Speech Synthesis
server/main.py
Other components
- User Speech
- Google Gemini
- Piper TTS
Connections 16
- User Speech → INMP441 Mic: speaks to
README.md - INMP441 Mic → ESP32 Firmware: audio input
README.md - ESP32 Firmware → WebSocket Link: streams audio
README.md - WebSocket Link → Client Handler: delivers audio
README.md - Client Handler → Audio Buffer: buffers audio
server/main.py - Client Handler → Audio Pipeline: invokes processing
server/main.py - Audio Pipeline → Speech Recognition: transcribes audio
server/main.py - Audio Pipeline → Reply Generation: requests reply
server/main.py - Reply Generation → Conversation History: uses context
server/main.py - Reply Generation → Google Gemini: generates response
server/main.py - Audio Pipeline → Speech Synthesis: synthesizes reply
server/main.py - Speech Synthesis → Piper TTS: uses voice model
server/main.py - Audio Pipeline → Audio Streaming: streams generated audio
server/main.py - Audio Streaming → WebSocket Link: sends audio chunks
server/main.py - WebSocket Link → ESP32 Firmware: returns audio
README.md - ESP32 Firmware → Speaker Output: plays audio
README.md