How zuyingwo-lenovo/open-agent-asr works
Aura AI Voice Agent is a Streamlit voice-conversation app: the user speaks, adaptive VAD segments speech, ASR produces text, an agent backend generates a response, and TTS returns spoken audio while the UI shows subtitles and avatar states. A separate comparison interface evaluates NeMo and ONNX ASR. The README documents additional provider choices and telemetry; unsampled implementation details are represented only where the tree or documentation supports them.
Main entry points: app_agent.py Voice agent UI · tts_engine.py Speech synthesis · hailo_llm_runner.py Hailo agent runner · hailo_llm_wrapper.cpp Native LLM wrapper
Components 18
Conversation interface
- Voice agent UI
app_agent.py - Character profiles
character_manager.py - ASR comparison UI
app.py
Audio capture and output
- Adaptive VAD
vad_engine.py - Speech synthesis
tts_engine.py
Speech recognition
- ONNX Nemotron ASR
nemotron_onnx.py - NeMo ASR
nemotron_nemo.py
Agent inference
- Hailo agent runner
hailo_llm_runner.py - Gemini agent
app_agent.py - Ollama provider
app_agent.py
NPU runtime
- Native LLM wrapper
csrc/hailo_llm_wrapper.cpp - NPU telemetry
csrc/hailo_llm_wrapper.cpp
Other components
- Speaker
- Browser user
- Microphone
- Speaker output
- Edge TTS service
- Hailo-10H NPU
Connections 27
- Speaker → Microphone: speaks
- Microphone → Adaptive VAD: captures audio
- Adaptive VAD → Voice agent UI: queues speech
- Voice agent UI → ONNX Nemotron ASR: transcribes
- ONNX Nemotron ASR → Voice agent UI: returns text
- Voice agent UI → Hailo agent runner: requests response
- Hailo agent runner → Native LLM wrapper: calls native API
- Native LLM wrapper → Hailo-10H NPU: runs inference
- Hailo-10H NPU → Native LLM wrapper: streams tokens
- Native LLM wrapper → Hailo agent runner: returns tokens
- Hailo agent runner → Voice agent UI: streams response
- Voice agent UI → Speech synthesis: synthesizes reply
- Speech synthesis → Speaker output: plays audio
- Voice agent UI → Adaptive VAD: mutes during playback
- Browser user → Voice agent UI: uses interface
- Voice agent UI → Character profiles: loads persona
- Voice agent UI → Gemini agent: selects provider
- Voice agent UI → Ollama provider: selects provider
- Gemini agent → Voice agent UI: returns response
- Ollama provider → Voice agent UI: returns response
- Speech synthesis → Edge TTS service: requests synthesis
- Edge TTS service → Speech synthesis: returns audio
- Voice agent UI → NPU telemetry: reads metrics
- NPU telemetry → Hailo-10H NPU: queries device
- Browser user → ASR comparison UI: runs comparison
- ASR comparison UI → ONNX Nemotron ASR: transcribes audio
- ASR comparison UI → NeMo ASR: transcribes audio