How upxill/assemblyai-voice-agent works
This is a browser-driven realtime voice agent: the browser captures and streams microphone audio directly to AssemblyAI, then sends completed utterances to the FastAPI backend for a LangGraph conversation and speaks replies with browser TTS. The backend separately mints the short-lived STT token. Agent tools and server-side conversation memory support the conversation stage. The graph combines shown source relationships with README-documented flows; unsampled UI markup and external-service details are represented only where documented.
Main entry points: app.js Audio orchestration · api.py HTTP routes · agent.py LangGraph agent · main.py FastAPI assembly
Components 16
Browser voice client
- Voice interface
app/static/index.html - Audio orchestration
app/static/app.js - PCM downsampler
app/static/pcm-worklet.js - Speech synthesis
HTTP backend
- FastAPI assembly
app/main.py - HTTP routes
app/api.py - Chat validation
app/schemas.py - Runtime settings
app/settings.py - STT token minting
app/assemblyai.py
Conversation agent
- LangGraph agent
app/agent.py - Conversation memory
app/agent.py - Agent tools
app/tools.py
External services
- Realtime STT
- LLM Gateway
- AssemblyAI token API
Other components
- Voice user
Connections 19
- Voice user → Voice interface: uses
README.md - Voice interface → Audio orchestration: hosts
README.md - Audio orchestration → PCM downsampler: loads worklet
app/static/app.js - Audio orchestration → Realtime STT: streams audio
README.md - Realtime STT → Audio orchestration: sends turns
README.md - Audio orchestration → HTTP routes: fetches config
app/static/app.js - Audio orchestration → HTTP routes: requests token
app/static/app.js - Audio orchestration → HTTP routes: posts turns
app/static/app.js - Audio orchestration → Speech synthesis: speaks replies
README.md - FastAPI assembly → HTTP routes: registers routes
app/main.py - FastAPI assembly → Runtime settings: reads static path
app/main.py - HTTP routes → Chat validation: validates chat
app/api.py - HTTP routes → Runtime settings: reads settings
app/api.py - HTTP routes → STT token minting: mints token
app/api.py - HTTP routes → LangGraph agent: runs turn
app/api.py - STT token minting → AssemblyAI token API: requests token
app/assemblyai.py - LangGraph agent → Agent tools: dispatches tools
app/agent.py - LangGraph agent → Conversation memory: loads and saves
app/agent.py - LangGraph agent → LLM Gateway: invokes model
README.md