# zuyingwo-lenovo/open-agent-asr architecture

> Architecture diagram of the GitHub repository zuyingwo-lenovo/open-agent-asr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zuyingwo-lenovo/open-agent-asr
- Repository: https://github.com/zuyingwo-lenovo/open-agent-asr

## Overview

Aura AI Voice Agent is a Streamlit voice-conversation app: the user speaks, adaptive VAD segments speech, ASR produces text, an agent backend generates a response, and TTS returns spoken audio while the UI shows subtitles and avatar states. A separate comparison interface evaluates NeMo and ONNX ASR. The README documents additional provider choices and telemetry; unsampled implementation details are represented only where the tree or documentation supports them.

Main entry points: [`app_agent.py`](https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/app_agent.py) (Voice agent UI), [`tts_engine.py`](https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/tts_engine.py) (Speech synthesis), [`hailo_llm_runner.py`](https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/hailo_llm_runner.py) (Hailo agent runner), [`csrc/hailo_llm_wrapper.cpp`](https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/csrc/hailo_llm_wrapper.cpp) (Native LLM wrapper)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interaction["Conversation interface"]
  node_app_agent["Voice agent UI<br/>[app_agent.py]"]
  node_character["Character profiles"]
  node_app_compare["ASR comparison UI<br/>[app.py]"]
end

subgraph group_audio["Audio capture and output"]
  node_vad["Adaptive VAD<br/>[vad_engine.py]"]
  node_tts["Speech synthesis<br/>[tts_engine.py]"]
end

subgraph group_recognition["Speech recognition"]
  node_onnx["ONNX Nemotron ASR<br/>[nemotron_onnx.py]"]
  node_nemo["NeMo ASR<br/>[nemotron_nemo.py]"]
end

subgraph group_agent["Agent inference"]
  node_hailo_runner["Hailo agent runner"]
  node_gemini["Gemini agent<br/>[app_agent.py]"]
  node_ollama["Ollama provider<br/>[app_agent.py]"]
end

subgraph group_native["NPU runtime"]
  node_hailo_native["Native LLM wrapper"]
  node_telemetry["NPU telemetry"]
end

node_user(("Speaker"))
node_browser(("Browser user"))
node_mic(("Microphone"))
node_speaker(("Speaker output"))
node_edge_tts["Edge TTS service"]
node_npu{{"Hailo-10H NPU"}}

node_user -->|"speaks"| node_mic
node_mic -->|"captures audio"| node_vad
node_vad -->|"queues speech"| node_app_agent
node_app_agent -->|"transcribes"| node_onnx
node_onnx -->|"returns text"| node_app_agent
node_app_agent -->|"requests response"| node_hailo_runner
node_hailo_runner -->|"calls native API"| node_hailo_native
node_hailo_native -->|"runs inference"| node_npu
node_npu -->|"streams tokens"| node_hailo_native
node_hailo_native -->|"returns tokens"| node_hailo_runner
node_hailo_runner -->|"streams response"| node_app_agent
node_app_agent -->|"synthesizes reply"| node_tts
node_tts -->|"plays audio"| node_speaker
node_app_agent -->|"mutes during playback"| node_vad
node_browser -->|"uses interface"| node_app_agent
node_app_agent -->|"loads persona"| node_character
node_app_agent -.->|"selects provider"| node_gemini
node_app_agent -.->|"selects provider"| node_ollama
node_gemini -.->|"returns response"| node_app_agent
node_ollama -.->|"returns response"| node_app_agent
node_tts -.->|"requests synthesis"| node_edge_tts
node_edge_tts -.->|"returns audio"| node_tts
node_app_agent -->|"reads metrics"| node_telemetry
node_telemetry -->|"queries device"| node_npu
node_browser -->|"runs comparison"| node_app_compare
node_app_compare -->|"transcribes audio"| node_onnx
node_app_compare -->|"transcribes audio"| node_nemo

click node_app_agent "https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/app_agent.py"
click node_character "https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/character_manager.py"
click node_app_compare "https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/app.py"
click node_vad "https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/vad_engine.py"
click node_tts "https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/tts_engine.py"
click node_onnx "https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/nemotron_onnx.py"
click node_nemo "https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/nemotron_nemo.py"
click node_hailo_runner "https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/hailo_llm_runner.py"
click node_gemini "https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/app_agent.py"
click node_ollama "https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/app_agent.py"
click node_hailo_native "https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/csrc/hailo_llm_wrapper.cpp"
click node_telemetry "https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/csrc/hailo_llm_wrapper.cpp"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app_agent,node_character,node_app_compare,node_browser toneBlue
class node_vad,node_tts toneAmber
class node_onnx,node_nemo toneMint
class node_hailo_runner,node_gemini,node_ollama toneRose
class node_hailo_native,node_telemetry,node_user,node_mic,node_speaker,node_edge_tts,node_npu toneIndigo
```

## Components

### Conversation interface

- [Voice agent UI](https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/app_agent.py): `app_agent.py`
- [Character profiles](https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/character_manager.py): `character_manager.py`
- [ASR comparison UI](https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/app.py): `app.py`

### Audio capture and output

- [Adaptive VAD](https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/vad_engine.py): `vad_engine.py`
- [Speech synthesis](https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/tts_engine.py): `tts_engine.py`

### Speech recognition

- [ONNX Nemotron ASR](https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/nemotron_onnx.py): `nemotron_onnx.py`
- [NeMo ASR](https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/nemotron_nemo.py): `nemotron_nemo.py`

### Agent inference

- [Hailo agent runner](https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/hailo_llm_runner.py): `hailo_llm_runner.py`
- [Gemini agent](https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/app_agent.py): `app_agent.py`
- [Ollama provider](https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/app_agent.py): `app_agent.py`

### NPU runtime

- [Native LLM wrapper](https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/csrc/hailo_llm_wrapper.cpp): `csrc/hailo_llm_wrapper.cpp`
- [NPU telemetry](https://github.com/zuyingwo-lenovo/open-agent-asr/blob/develop/hailo-10h/csrc/hailo_llm_wrapper.cpp): `csrc/hailo_llm_wrapper.cpp`

### Other components

- **Speaker**
- **Browser user**
- **Microphone**
- **Speaker output**
- **Edge TTS service**
- **Hailo-10H NPU**

## Connections

- Speaker → Microphone: speaks
- Microphone → Adaptive VAD: captures audio
- Adaptive VAD → Voice agent UI: queues speech
- Voice agent UI → ONNX Nemotron ASR: transcribes
- ONNX Nemotron ASR → Voice agent UI: returns text
- Voice agent UI → Hailo agent runner: requests response
- Hailo agent runner → Native LLM wrapper: calls native API
- Native LLM wrapper → Hailo-10H NPU: runs inference
- Hailo-10H NPU → Native LLM wrapper: streams tokens
- Native LLM wrapper → Hailo agent runner: returns tokens
- Hailo agent runner → Voice agent UI: streams response
- Voice agent UI → Speech synthesis: synthesizes reply
- Speech synthesis → Speaker output: plays audio
- Voice agent UI → Adaptive VAD: mutes during playback
- Browser user → Voice agent UI: uses interface
- Voice agent UI → Character profiles: loads persona
- Voice agent UI → Gemini agent: selects provider
- Voice agent UI → Ollama provider: selects provider
- Gemini agent → Voice agent UI: returns response
- Ollama provider → Voice agent UI: returns response
- Speech synthesis → Edge TTS service: requests synthesis
- Edge TTS service → Speech synthesis: returns audio
- Voice agent UI → NPU telemetry: reads metrics
- NPU telemetry → Hailo-10H NPU: queries device
- Browser user → ASR comparison UI: runs comparison
- ASR comparison UI → ONNX Nemotron ASR: transcribes audio
- ASR comparison UI → NeMo ASR: transcribes audio