How learnedvector/a-hackers-ai-voice-assistant works
This repository is a collection of separate ML subsystems for a voice assistant, plus some shared orchestration glue. It is not a single monolithic application; most of the code is organized around building, training, optimizing, and demoing three core audio models:
• wake word detection
• speech recognition
• natural language understanding
There is also a placeholder package for speech synthesis, and a top-level core.py that appears intended to integrate the subsystems into a full assistant, though the README says that end-to-end assistant logic is still a TODO.
High-level architecture
The repo is organized into four main functional boundaries:
1. Wake word subsystem
Detects when the assistant should start listening.
2. Speech recognition subsystem
Converts spoken audio into text.
3. NLU subsystem
Classifies or interprets text into intents / labels.
4. Speech synthesis subsystem
Present as a package boundary, but not implemented in the file tree shown.
A fifth boundary cuts across all of them:
5. Training / data / model optimization toolchain
Each subsystem has its own dataset code, training script, and graph optimization step so models can move from research training to inference/demo usage.
Top-level repo layout
• VoiceAssistant/core.py
Likely the central integration point for the assistant pipeline. Based on the README, this is intended to connect wake word → speech recognition → NLU → synthesis → skills.
• VoiceAssistant/wakeword/
Wake-word detection implementation and training utilities.
• VoiceAssistant/speechrecognition/
Speech-to-text implementation, including a browser/web demo.
• VoiceAssistant/nlu/
Natural language understanding implementation and datasets.
• VoiceAssistant/speechsynthesis/
Package exists, but there are no model or engine files visible here yet.
• fun/arnold_audio/
Sample audio files used for experimentation/demo content, not part of the assistant pipeline.
• Dockerfile, cpu.Dockerfile
Container definitions for GPU/CUDA and CPU-only environments.
• requirements.txt, rpi-requirements.txt
Python dependency sets for general and Raspberry Pi-related environments.
Wake word subsystem
Path: VoiceAssistant/wakeword/
Purpose
This subsystem detects a trigger phrase so the assistant knows when to begin processing audio.
Main components
• engine.py
Inference/demo entry point for wake word detection.
• neuralnet/
- dataset.py — dataset loading and preprocessing
- model.py — network definition
- train.py — model training
- optimize_graph.py — exports an inference-ready optimized model
- __init__.py — package boundary
• scripts/
- collect_wakeword_audio.py — audio capture for wake word and background noise
- create_wakeword_jsons.py — converts audio files into training/test manifests
- split_audio_into_chunks.py — chops recordings into fixed-length segments
- split_commonvoice.py — prepares Common Voice audio into chunks
- replicate_audios.py — duplicates wake word samples to address class imbalance
Data flow
The wake word workflow is:
1. collect or prepare labeled audio,
2. create JSON manifests,
3. train the neural net,
4. optimize/export the graph,
5. run inference in engine.py.
Architectural notes
This subsystem is a self-contained binary classification pipeline:
• label 1 = wake word
• label 0 = non-wake/background speech
It relies heavily on pre-segmented audio clips and dataset balancing. The scripts indicate the system expects fixed-duration input windows.
Speech recognition subsystem
Path: VoiceAssistant/speechrecognition/
Purpose
This subsystem converts audio into text. The README describes it as a pretrained/fine-tunable ASR model.
Main components
• engine.py
Offline inference/demo entry point for speech recognition.
• decoder.py
Post-processing / decoding logic for model outputs, likely CTC decoding.
• neuralnet/
- dataset.py — audio/text dataset handling
- model.py — ASR model definition
- scorer.py — scoring/evaluation support
- train.py — training and fine-tuning
- optimize_graph.py — exports an inference-ready graph
- utils.py — helper functions
- __init__.py — package boundary
• demo/
- demo.py — demo app launcher
- templates/demo.html — web UI for the demo
• scripts/
- mimic_create_jsons.py — builds train/test manifests from Mimic Recording Studio output
- commonvoice_create_jsons.py — converts Common Voice data and builds manifests
Data flow
The speech recognition path is:
1. ingest audio-text pairs from Mimic Recording Studio or Mozilla Common Voice,
2. optionally convert/normalize audio files,
3. generate train/validation/test JSON manifests,
4. train or fine-tune the ASR model,
5. optimize/export the model,
6. run inference through engine.py or the web demo.
Architectural notes
This subsystem has the richest inference support:
• a standalone engine for direct testing,
• a demo app with HTML templates for interactive use,
• a decoder layer separating model output from text generation.
The README mentions CTC decoding dependencies, so the architecture is likely based on a sequence model that produces token probabilities, then decodes them into text.
NLU subsystem
Path: VoiceAssistant/nlu/
Purpose
This subsystem interprets recognized text into intent/label predictions. The README labels this as natural language understanding, but the actual functionality appears to be a text classification or intent detection model.
Main components
• engine.py
Inference/demo entry point for NLU.
• meta_data.bin
Serialized metadata/artifacts used by the NLU model, likely vocabulary, label mappings, or preprocessing state.
• neuralnet/
- config.py — model/configuration settings
- dataset.py — dataset preparation
- model.py — model definition
- train.py — training
- optimize_graph.py — export/optimize for inference
- utils.py — helpers
- __init__.py
• scripts/
- make_dataset.py — dataset creation from source text examples
• nlu_dataset/
- er_dataset.csv
- is_dataset.csv
- nlu.csv
- nlucopy.csv
Data flow
The NLU flow is:
1. curate labeled text examples into CSV/dataset files,
2. build training data and metadata,
3. train the classifier,
4. optimize/export the graph,
5. run inference in engine.py.
Architectural notes
This subsystem is text-only and sits downstream of speech recognition. Its dependency on meta_data.bin suggests a stable preprocessing contract between training and inference, such as tokenization or label mapping.
Speech synthesis subsystem
Path: VoiceAssistant/speechsynthesis/
Purpose
This package boundary exists, but there are no visible engine, model, or training files in the provided tree.
Architectural role
The README explicitly lists speech synthesis as planned, not completed. In a full assistant pipeline, this subsystem would be the final stage that turns NLU/skills outputs into audio responses.
Core assistant integration boundary
Path: VoiceAssistant/core.py
Purpose
This is likely intended to be the top-level coordinator that wires together:
• wake word detection,
• speech recognition,
• NLU,
• speech synthesis,
• skills/action execution.
Architectural role
The README says this integration is still a TODO, so this is more of a future system boundary than a fully realized runtime. In diagram terms, it should be shown as the orchestration layer above the individual ML services/modules.
Training vs inference separation
A major architectural pattern across the repo is the split between:
• training-time code
• inference-time code
• model export/optimization code
This pattern appears in wake word, speech recognition, and NLU.
Typical module roles:
• dataset.py — prepares examples and batching
• model.py — defines the network
• train.py — trains from labeled data
• optimize_graph.py — converts the trained model to an inference-friendly artifact
• engine.py — loads the optimized artifact for demo/runtime use
This separation is important for an architecture diagram because it shows that the repo is not just one application runtime; it also includes the full ML lifecycle.
External datasets and tooling
These materially affect the architecture:
• Mozilla Common Voice
Used for speech recognition and wake word data preparation.
• Mimic Recording Studio
Used for collecting personalized speech recognition data.
• PyTorch
The core ML framework used across the neural network packages.
• CTC decoding library (ctcdecode)
Specifically relevant to speech recognition output decoding.
• PortAudio / PyAudio
Needed for microphone/audio capture in recording and demo workflows.
• Docker
Two build paths exist:
- GPU/CUDA container via Dockerfile
- CPU-only container via cpu.Dockerfile
These choices imply the repo is designed to run both as a local development environment and as a containerized ML system.
Boundaries and dependencies between subsystems
The intended assistant pipeline is:
1. Wake word engine listens continuously for activation.
2. Once triggered, speech recognition processes the user’s utterance.
3. The recognized text is passed to NLU for intent interpretation.
4. A future skills layer would decide what action to take.
5. Speech synthesis would generate the spoken response.
At present, the repo is more complete on the left side of that pipeline than the right:
• wake word: implemented
• speech recognition: implemented
• NLU: implemented
• synthesis/skills/core orchestration: scaffolded or planned
Practical diagramming guidance
If you are drawing an architecture diagram for this repo, the cleanest representation is:
• one top-level “VoiceAssistant” application boundary,
• three major ML subsystems inside it,
• each subsystem split into training/data/inference artifacts,
• a central orchestration layer in core.py,
• external dependencies feeding into training and runtime capture,
• Docker and dependency files shown as environment/runtime support, not as core logic.
The most accurate summary is that this repository is an ML system workspace for building a voice assistant, with independent pipelines for wake word detection, ASR, and NLU, but without a finished end-to-end assistant runtime yet.
Main entry points: core.py Core · speechrecognition/engine.py ASR engine · nlu/engine.py NLU engine · wakeword/engine.py Wake engine
Components 22
VoiceAssistant
Top-level assistant workspace and ML subsystems
- Core · Intended top-level integration layer for the assistant pipeline
VoiceAssistant/core.py - Synthesis pkg · Speech synthesis boundary is present but not implemented
VoiceAssistant/speechsynthesis/__init__.py
Wake word
Trigger detection pipeline
- Wake engine · Runs wake-word detection at runtime
VoiceAssistant/wakeword/engine.py - Wake data · Loads and prepares fixed-window audio examples
VoiceAssistant/wakeword/neuralnet/dataset.py - Wake model · Neural network for wake-word vs background detection
VoiceAssistant/wakeword/neuralnet/model.py - Wake train · Trains the wake-word model from labeled audio
VoiceAssistant/wakeword/neuralnet/train.py - Wake export · Optimizes the trained wake-word model for inference
VoiceAssistant/wakeword/neuralnet/optimize_graph.py
Speech recognition
Audio-to-text pipeline
- ASR engine · Runs offline speech recognition
VoiceAssistant/speechrecognition/engine.py - Decoder · Converts model outputs into text
VoiceAssistant/speechrecognition/decoder.py - ASR data · Builds audio-text training batches and manifests
VoiceAssistant/speechrecognition/neuralnet/dataset.py - ASR model · Speech recognition network
VoiceAssistant/speechrecognition/neuralnet/model.py - ASR train · Trains and fine-tunes the ASR model
VoiceAssistant/speechrecognition/neuralnet/train.py - ASR export · Optimizes the ASR model for inference/demo use
VoiceAssistant/speechrecognition/neuralnet/optimize_graph.py - ASR demo · Browser demo entry point and UI
VoiceAssistant/speechrecognition/demo/demo.py
NLU
Text intent classification pipeline
- NLU engine · Runs intent or label prediction over text
VoiceAssistant/nlu/engine.py - Metadata · Serialized preprocessing and label metadata
VoiceAssistant/nlu/meta_data.bin - NLU data · Curated labeled text examples
VoiceAssistant/nlu/nlu_dataset/nlu.csv - NLU model · Model for intent or label classification
VoiceAssistant/nlu/neuralnet/model.py - NLU train · Trains the NLU classifier
VoiceAssistant/nlu/neuralnet/train.py - NLU export · Optimizes the NLU model for inference
VoiceAssistant/nlu/neuralnet/optimize_graph.py
Support
Packaging, demo assets, and environment support
- Env support · Container and dependency setup for CPU, GPU, and Raspberry Pi environments
Dockerfile - Audio samples · Example audio clips used for experimentation and fun demos
fun/arnold_audio/Hi.wav
Connections 23
- Core → Wake engine: starts · Coordinator begins with wake-word listening
- Core → ASR engine: passes audio · Coordinator routes captured speech to ASR
- Core → NLU engine: passes text · Coordinator forwards recognized text to NLU
- Core → Synthesis pkg: responds via · Future response generation boundary
- Wake engine → Wake model: loads · Inference uses optimized wake-word network
- Wake model → Wake export: exported from · Inference artifact is produced by graph optimization
- Wake train → Wake model: trains · Training produces the wake-word network
- Wake data → Wake train: feeds · Prepared audio windows supply training data
- Wake export → Wake engine: serves · Optimized graph is consumed by runtime engine
- ASR engine → Decoder: decodes through · Model outputs are post-processed into text
- ASR engine → ASR model: loads · Inference uses the ASR network
- ASR model → ASR export: exported from · Inference artifact comes from optimization step
- ASR train → ASR model: trains · Training and fine-tuning produce the ASR network
- ASR data → ASR train: feeds · Audio-text corpora feed training
- ASR export → ASR engine: serves · Optimized ASR graph powers runtime inference
- ASR demo → ASR engine: uses · Web demo wraps the ASR inference path
- NLU engine → Metadata: loads · Runtime relies on saved preprocessing and labels
- NLU engine → NLU model: loads · Inference uses the NLU classifier
- NLU train → NLU model: trains · Training produces the text model
- NLU data → NLU train: feeds · Labeled examples feed classifier training
- NLU export → NLU engine: serves · Optimized graph powers inference
- Env support → Core: supports · Container and dependency setup support execution
- Audio samples → Wake engine: demo input · Sample clips support experimentation