How dhruthi9701/speech-recognition-with-esp32-using-vosk-model works
Project Type & Purpose
• This is a hybrid embedded/host‐based system:
– Host side: Python wake-word listener using Vosk.
– Embedded side: ESP32 + DFPlayer Mini handle MP3 playback and power management.
• Purpose: Listen for keywords (“jarvis”, “hello”, etc.) on a PC microphone, send a trigger byte over serial to the ESP32, play a matching MP3 from microSD, optionally deep-sleep.
Main Components
1. Hardware
– Microphone (PC-side)
– PC (runs Python + Vosk model)
– USB/Serial link
– ESP32 Dev Board (UART2 on GPIO18/19)
– DFPlayer Mini MP3 Module
– microSD Card (contains 0001.mp3–0005.mp3)
– Speaker
2. Software & Firmware
– Python scripts:
• mic_list.py, mic_rms.py, test_serial.py (utilities)
• wakeword_vosk_to_esp32.py (core listener & serial sender)
• Vosk model folder (vosk-model-small-en-us-0.15)
– Arduino sketch (esp32_to_dfplayer.ino) using DFRobotDFPlayerMini library
Relationships & Data Flows
1. Audio Input → Vosk
• Mic captures live audio → sounddevice → Vosk model → triggers keyword detection
2. Trigger Byte over Serial
• Python sends one of {0x01…0x05} via pyserial to ESP32 USB port
3. ESP32 Command Handling
• UART2 RX receives byte → switch statement → DFRobotDFPlayerMini.play(trackIndex)
• If byte==0x05, play track 5 then call esp_deep_sleep()
4. Playback
• DFPlayer reads MP3 from microSD → outputs audio to speaker
5. Deep-Sleep (on “hammer time”)
• ESP32 enters low-power mode after playing track 5
Architectural Patterns & Principles
• Clear separation of concerns:
– PC host: compute-heavy speech recognition
– Embedded: real-time playback and power management
• Event-driven control flow (keyword → trigger → action)
• Serial (UART) as boundary/interface between host and device
Technologies & Frameworks
• Python 3 with vosk, sounddevice, pyserial
• Vosk offline speech model
• Arduino framework for ESP32
• DFRobotDFPlayerMini Arduino library
– Note on deep-sleep behavior