How getstream/vision-agents works
Vision Agents is a framework for low-latency multimodal conversational agents. The principal workflow accepts live audio/video through an edge transport, runs inference and optional video processing, then returns generated speech, text, or avatar media. Its core also supports turn detection, tools/MCP, retrieval, session memory, telephony, and production serving. Provider integrations are separate plugins. README documents several capabilities whose internal wiring is not established by the sampled source, so those branches are shown without speculative call chains.
Main entry points: realtime_flow.py Inference flows · agents.py Agent lifecycle · llm.py LLM interface · edge_transport.py Edge transport
Components 26
Entry and serving
Agent runtime
- Agent lifecycle
agents-core/vision_agents/core/agents/agents.py - Edge transport
agents-core/vision_agents/core/edge/edge_transport.py - Inference flows
agents-core/vision_agents/core/agents/inference/realtime_flow.py - Video processors
agents-core/vision_agents/core/processors/base_processor.py - LLM interface
agents-core/vision_agents/core/llm/llm.py - Speech services
agents-core/vision_agents/core/stt/stt.py
Conversation capabilities
- Turn detection
agents-core/vision_agents/core/turn_detection/turn_detection.py - Tools and MCP
agents-core/vision_agents/core/mcp/mcp_manager.py - RAG retrieval
agents-core/vision_agents/core/rag/rag.py - Transcript
agents-core/vision_agents/core/agents/transcript/store.py - Avatar sync
agents-core/vision_agents/core/avatars/av_synchronizer.py
Provider plugins
- Phone channels
plugins/twilio/vision_agents/plugins/twilio/media_stream.py - Model integrations
plugins/openai/vision_agents/plugins/openai/openai_realtime.py - Vision integrations
plugins/ultralytics/vision_agents/plugins/ultralytics/yolo_pose_processor.py - Avatar integrations
plugins/lemonslice/vision_agents/plugins/lemonslice/lemonslice_rtc_manager.py
State and observability
- Session registry
agents-core/vision_agents/core/agents/session_registry/registry.py - Metrics and events
agents-core/vision_agents/core/observability/metrics.py
Other components
- Developer / caller
- Participant
- Video edge network
- Model provider
- Vector search service
- Stream Chat
Connections 26
- Developer / caller → CLI: runs commands
- Developer / caller → HTTP API: configures service
- HTTP API → Runner: dispatches requests
- Runner → Agent lifecycle: launches agent
- Participant → Video edge network: sends media
- Video edge network → Edge transport: delivers tracks
- Edge transport → Agent lifecycle: provides session
- Agent lifecycle → Inference flows: starts flow
- Inference flows → Video processors: processes video
- Inference flows → LLM interface: requests generation
- Inference flows → Speech services: transcribes and speaks
- Inference flows → Turn detection: detects turns
- Inference flows → Transcript: stores transcript
- LLM interface → Tools and MCP: dispatches tool calls
- Tools and MCP → Model provider: calls MCP services
- Inference flows → RAG retrieval: retrieves context
- RAG retrieval → Vector search service: searches vectors
- Agent lifecycle → Session registry: manages sessions
- Inference flows → Avatar sync: synchronizes media
- Avatar sync → Avatar integrations: uses avatar transport
- Phone channels → Edge transport: streams call media
- LLM interface → Model integrations: implemented by
- Video processors → Vision integrations: implemented by
- Model integrations → Model provider: sends inference
- Agent lifecycle → Metrics and events: reports metrics
- Transcript → Stream Chat: shares memory