# upxill/assemblyai-voice-agent architecture

> Architecture diagram of the GitHub repository upxill/assemblyai-voice-agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/upxill/assemblyai-voice-agent
- Repository: https://github.com/upxill/assemblyai-voice-agent

## Overview

This is a browser-driven realtime voice agent: the browser captures and streams microphone audio directly to AssemblyAI, then sends completed utterances to the FastAPI backend for a LangGraph conversation and speaks replies with browser TTS. The backend separately mints the short-lived STT token. Agent tools and server-side conversation memory support the conversation stage. The graph combines shown source relationships with README-documented flows; unsampled UI markup and external-service details are represented only where documented.

Main entry points: [`app/static/app.js`](https://github.com/upxill/assemblyai-voice-agent/blob/main/app/static/app.js) (Audio orchestration), [`app/api.py`](https://github.com/upxill/assemblyai-voice-agent/blob/main/app/api.py) (HTTP routes), [`app/agent.py`](https://github.com/upxill/assemblyai-voice-agent/blob/main/app/agent.py) (LangGraph agent), [`app/main.py`](https://github.com/upxill/assemblyai-voice-agent/blob/main/app/main.py) (FastAPI assembly)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_browser["Browser voice client"]
  node_ui["Voice interface<br/>[index.html]"]
  node_client["Audio orchestration<br/>[app.js]"]
  node_pcm["PCM downsampler<br/>[pcm-worklet.js]"]
  node_tts["Speech synthesis"]
end

subgraph group_backend["HTTP backend"]
  node_app["FastAPI assembly<br/>[main.py]"]
  node_routes["HTTP routes<br/>[api.py]"]
  node_request["Chat validation<br/>[schemas.py]"]
  node_config["Runtime settings<br/>[settings.py]"]
  node_token["STT token minting<br/>[assemblyai.py]"]
end

subgraph group_agent["Conversation agent"]
  node_agent["LangGraph agent<br/>[agent.py]"]
  node_memory[("Conversation memory<br/>[agent.py]")]
  node_tools["Agent tools<br/>[tools.py]"]
end

subgraph group_external["External services"]
  node_stt["Realtime STT"]
  node_gateway["LLM Gateway"]
  node_assembly["AssemblyAI token API"]
end

node_user(("Voice user"))

node_user -->|"uses"| node_ui
node_ui -->|"hosts"| node_client
node_client -->|"loads worklet"| node_pcm
node_client -->|"streams audio"| node_stt
node_stt -->|"sends turns"| node_client
node_client -->|"fetches config"| node_routes
node_client -->|"requests token"| node_routes
node_client -->|"posts turns"| node_routes
node_client -->|"speaks replies"| node_tts
node_app -->|"registers routes"| node_routes
node_app -->|"reads static path"| node_config
node_routes -->|"validates chat"| node_request
node_routes -->|"reads settings"| node_config
node_routes -->|"mints token"| node_token
node_routes -->|"runs turn"| node_agent
node_token -->|"requests token"| node_assembly
node_agent -->|"dispatches tools"| node_tools
node_agent -->|"loads and saves"| node_memory
node_agent -->|"invokes model"| node_gateway

click node_ui "https://github.com/upxill/assemblyai-voice-agent/blob/main/app/static/index.html"
click node_client "https://github.com/upxill/assemblyai-voice-agent/blob/main/app/static/app.js"
click node_pcm "https://github.com/upxill/assemblyai-voice-agent/blob/main/app/static/pcm-worklet.js"
click node_app "https://github.com/upxill/assemblyai-voice-agent/blob/main/app/main.py"
click node_routes "https://github.com/upxill/assemblyai-voice-agent/blob/main/app/api.py"
click node_request "https://github.com/upxill/assemblyai-voice-agent/blob/main/app/schemas.py"
click node_config "https://github.com/upxill/assemblyai-voice-agent/blob/main/app/settings.py"
click node_token "https://github.com/upxill/assemblyai-voice-agent/blob/main/app/assemblyai.py"
click node_agent "https://github.com/upxill/assemblyai-voice-agent/blob/main/app/agent.py"
click node_memory "https://github.com/upxill/assemblyai-voice-agent/blob/main/app/agent.py"
click node_tools "https://github.com/upxill/assemblyai-voice-agent/blob/main/app/tools.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ui,node_client,node_pcm,node_tts,node_user toneBlue
class node_app,node_routes,node_request,node_config,node_token toneAmber
class node_agent,node_memory,node_tools toneMint
class node_stt,node_gateway,node_assembly toneRose
```

## Components

### Browser voice client

- [Voice interface](https://github.com/upxill/assemblyai-voice-agent/blob/main/app/static/index.html): `app/static/index.html`
- [Audio orchestration](https://github.com/upxill/assemblyai-voice-agent/blob/main/app/static/app.js): `app/static/app.js`
- [PCM downsampler](https://github.com/upxill/assemblyai-voice-agent/blob/main/app/static/pcm-worklet.js): `app/static/pcm-worklet.js`
- **Speech synthesis**

### HTTP backend

- [FastAPI assembly](https://github.com/upxill/assemblyai-voice-agent/blob/main/app/main.py): `app/main.py`
- [HTTP routes](https://github.com/upxill/assemblyai-voice-agent/blob/main/app/api.py): `app/api.py`
- [Chat validation](https://github.com/upxill/assemblyai-voice-agent/blob/main/app/schemas.py): `app/schemas.py`
- [Runtime settings](https://github.com/upxill/assemblyai-voice-agent/blob/main/app/settings.py): `app/settings.py`
- [STT token minting](https://github.com/upxill/assemblyai-voice-agent/blob/main/app/assemblyai.py): `app/assemblyai.py`

### Conversation agent

- [LangGraph agent](https://github.com/upxill/assemblyai-voice-agent/blob/main/app/agent.py): `app/agent.py`
- [Conversation memory](https://github.com/upxill/assemblyai-voice-agent/blob/main/app/agent.py): `app/agent.py`
- [Agent tools](https://github.com/upxill/assemblyai-voice-agent/blob/main/app/tools.py): `app/tools.py`

### External services

- **Realtime STT**
- **LLM Gateway**
- **AssemblyAI token API**

### Other components

- **Voice user**

## Connections

- Voice user → Voice interface: uses (evidence: `README.md`)
- Voice interface → Audio orchestration: hosts (evidence: `README.md`)
- Audio orchestration → PCM downsampler: loads worklet (evidence: `app/static/app.js`)
- Audio orchestration → Realtime STT: streams audio (evidence: `README.md`)
- Realtime STT → Audio orchestration: sends turns (evidence: `README.md`)
- Audio orchestration → HTTP routes: fetches config (evidence: `app/static/app.js`)
- Audio orchestration → HTTP routes: requests token (evidence: `app/static/app.js`)
- Audio orchestration → HTTP routes: posts turns (evidence: `app/static/app.js`)
- Audio orchestration → Speech synthesis: speaks replies (evidence: `README.md`)
- FastAPI assembly → HTTP routes: registers routes (evidence: `app/main.py`)
- FastAPI assembly → Runtime settings: reads static path (evidence: `app/main.py`)
- HTTP routes → Chat validation: validates chat (evidence: `app/api.py`)
- HTTP routes → Runtime settings: reads settings (evidence: `app/api.py`)
- HTTP routes → STT token minting: mints token (evidence: `app/api.py`)
- HTTP routes → LangGraph agent: runs turn (evidence: `app/api.py`)
- STT token minting → AssemblyAI token API: requests token (evidence: `app/assemblyai.py`)
- LangGraph agent → Agent tools: dispatches tools (evidence: `app/agent.py`)
- LangGraph agent → Conversation memory: loads and saves (evidence: `app/agent.py`)
- LangGraph agent → LLM Gateway: invokes model (evidence: `README.md`)