# getstream/vision-agents architecture

> Architecture diagram of the GitHub repository getstream/vision-agents, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/getstream/vision-agents
- Repository: https://github.com/getstream/vision-agents

## Overview

Vision Agents is a framework for low-latency multimodal conversational agents. The principal workflow accepts live audio/video through an edge transport, runs inference and optional video processing, then returns generated speech, text, or avatar media. Its core also supports turn detection, tools/MCP, retrieval, session memory, telephony, and production serving. Provider integrations are separate plugins. README documents several capabilities whose internal wiring is not established by the sampled source, so those branches are shown without speculative call chains.

Main entry points: [`agents-core/vision_agents/core/agents/inference/realtime_flow.py`](https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/agents/inference/realtime_flow.py) (Inference flows), [`agents-core/vision_agents/core/agents/agents.py`](https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/agents/agents.py) (Agent lifecycle), [`agents-core/vision_agents/core/llm/llm.py`](https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/llm/llm.py) (LLM interface), [`agents-core/vision_agents/core/edge/edge_transport.py`](https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/edge/edge_transport.py) (Edge transport)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry and serving"]
  node_cli["CLI<br/>[main.py]"]
  node_http["HTTP API<br/>[api.py]"]
  node_runner["Runner<br/>[runner.py]"]
end

subgraph group_runtime["Agent runtime"]
  node_agent["Agent lifecycle<br/>[agents.py]"]
  node_edge["Edge transport<br/>[edge_transport.py]"]
  node_inference["Inference flows<br/>[realtime_flow.py]"]
  node_processors["Video processors<br/>[base_processor.py]"]
  node_llm["LLM interface<br/>[llm.py]"]
  node_stt_tts["Speech services<br/>[stt.py]"]
end

subgraph group_conversation["Conversation capabilities"]
  node_turns["Turn detection<br/>[turn_detection.py]"]
  node_tools["Tools and MCP<br/>[mcp_manager.py]"]
  node_rag["RAG retrieval<br/>[rag.py]"]
  node_transcript["Transcript<br/>[store.py]"]
  node_avatars["Avatar sync<br/>[av_synchronizer.py]"]
end

subgraph group_integrations["Provider plugins"]
  node_telephony["Phone channels<br/>[media_stream.py]"]
  node_provider_plugins["Model integrations<br/>[openai_realtime.py]"]
  node_vision_plugins["Vision integrations"]
  node_avatar_plugins["Avatar integrations"]
end

subgraph group_operations["State and observability"]
  node_session_store[("Session registry<br/>[registry.py]")]
  node_observability["Metrics and events<br/>[metrics.py]"]
end

node_developer(("Developer / caller"))
node_user(("Participant"))
node_network["Video edge network"]
node_model_service["Model provider"]
node_vector_store[("Vector search service")]
node_chat_service["Stream Chat"]

node_developer -->|"runs commands"| node_cli
node_developer -->|"configures service"| node_http
node_http -->|"dispatches requests"| node_runner
node_runner -->|"launches agent"| node_agent
node_user -->|"sends media"| node_network
node_network -->|"delivers tracks"| node_edge
node_edge -->|"provides session"| node_agent
node_agent -->|"starts flow"| node_inference
node_inference -->|"processes video"| node_processors
node_inference -->|"requests generation"| node_llm
node_inference -->|"transcribes and speaks"| node_stt_tts
node_inference -->|"detects turns"| node_turns
node_inference -->|"stores transcript"| node_transcript
node_llm -->|"dispatches tool calls"| node_tools
node_tools -.->|"calls MCP services"| node_model_service
node_inference -.->|"retrieves context"| node_rag
node_rag -.->|"searches vectors"| node_vector_store
node_agent -->|"manages sessions"| node_session_store
node_inference -.->|"synchronizes media"| node_avatars
node_avatars -.->|"uses avatar transport"| node_avatar_plugins
node_telephony -.->|"streams call media"| node_edge
node_llm -.->|"implemented by"| node_provider_plugins
node_processors -.->|"implemented by"| node_vision_plugins
node_provider_plugins -.->|"sends inference"| node_model_service
node_agent -->|"reports metrics"| node_observability
node_transcript -.->|"shares memory"| node_chat_service

click node_cli "https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/cli/main.py"
click node_http "https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/runner/http/api.py"
click node_runner "https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/runner/runner.py"
click node_agent "https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/agents/agents.py"
click node_edge "https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/edge/edge_transport.py"
click node_inference "https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/agents/inference/realtime_flow.py"
click node_processors "https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/processors/base_processor.py"
click node_llm "https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/llm/llm.py"
click node_stt_tts "https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/stt/stt.py"
click node_turns "https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/turn_detection/turn_detection.py"
click node_tools "https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/mcp/mcp_manager.py"
click node_rag "https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/rag/rag.py"
click node_transcript "https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/agents/transcript/store.py"
click node_avatars "https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/avatars/av_synchronizer.py"
click node_telephony "https://github.com/getstream/vision-agents/blob/main/plugins/twilio/vision_agents/plugins/twilio/media_stream.py"
click node_provider_plugins "https://github.com/getstream/vision-agents/blob/main/plugins/openai/vision_agents/plugins/openai/openai_realtime.py"
click node_vision_plugins "https://github.com/getstream/vision-agents/blob/main/plugins/ultralytics/vision_agents/plugins/ultralytics/yolo_pose_processor.py"
click node_avatar_plugins "https://github.com/getstream/vision-agents/blob/main/plugins/lemonslice/vision_agents/plugins/lemonslice/lemonslice_rtc_manager.py"
click node_session_store "https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/agents/session_registry/registry.py"
click node_observability "https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/observability/metrics.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_http,node_runner toneBlue
class node_agent,node_edge,node_inference,node_processors,node_llm,node_stt_tts,node_vector_store toneAmber
class node_turns,node_tools,node_rag,node_transcript,node_avatars toneMint
class node_telephony,node_provider_plugins,node_vision_plugins,node_avatar_plugins toneRose
class node_session_store,node_observability,node_developer,node_user,node_network,node_model_service,node_chat_service toneIndigo
```

## Components

### Entry and serving

- [CLI](https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/cli/main.py): `agents-core/vision_agents/cli/main.py`
- [HTTP API](https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/runner/http/api.py): `agents-core/vision_agents/core/runner/http/api.py`
- [Runner](https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/runner/runner.py): `agents-core/vision_agents/core/runner/runner.py`

### Agent runtime

- [Agent lifecycle](https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/agents/agents.py): `agents-core/vision_agents/core/agents/agents.py`
- [Edge transport](https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/edge/edge_transport.py): `agents-core/vision_agents/core/edge/edge_transport.py`
- [Inference flows](https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/agents/inference/realtime_flow.py): `agents-core/vision_agents/core/agents/inference/realtime_flow.py`
- [Video processors](https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/processors/base_processor.py): `agents-core/vision_agents/core/processors/base_processor.py`
- [LLM interface](https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/llm/llm.py): `agents-core/vision_agents/core/llm/llm.py`
- [Speech services](https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/stt/stt.py): `agents-core/vision_agents/core/stt/stt.py`

### Conversation capabilities

- [Turn detection](https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/turn_detection/turn_detection.py): `agents-core/vision_agents/core/turn_detection/turn_detection.py`
- [Tools and MCP](https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/mcp/mcp_manager.py): `agents-core/vision_agents/core/mcp/mcp_manager.py`
- [RAG retrieval](https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/rag/rag.py): `agents-core/vision_agents/core/rag/rag.py`
- [Transcript](https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/agents/transcript/store.py): `agents-core/vision_agents/core/agents/transcript/store.py`
- [Avatar sync](https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/avatars/av_synchronizer.py): `agents-core/vision_agents/core/avatars/av_synchronizer.py`

### Provider plugins

- [Phone channels](https://github.com/getstream/vision-agents/blob/main/plugins/twilio/vision_agents/plugins/twilio/media_stream.py): `plugins/twilio/vision_agents/plugins/twilio/media_stream.py`
- [Model integrations](https://github.com/getstream/vision-agents/blob/main/plugins/openai/vision_agents/plugins/openai/openai_realtime.py): `plugins/openai/vision_agents/plugins/openai/openai_realtime.py`
- [Vision integrations](https://github.com/getstream/vision-agents/blob/main/plugins/ultralytics/vision_agents/plugins/ultralytics/yolo_pose_processor.py): `plugins/ultralytics/vision_agents/plugins/ultralytics/yolo_pose_processor.py`
- [Avatar integrations](https://github.com/getstream/vision-agents/blob/main/plugins/lemonslice/vision_agents/plugins/lemonslice/lemonslice_rtc_manager.py): `plugins/lemonslice/vision_agents/plugins/lemonslice/lemonslice_rtc_manager.py`

### State and observability

- [Session registry](https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/agents/session_registry/registry.py): `agents-core/vision_agents/core/agents/session_registry/registry.py`
- [Metrics and events](https://github.com/getstream/vision-agents/blob/main/agents-core/vision_agents/core/observability/metrics.py): `agents-core/vision_agents/core/observability/metrics.py`

### Other components

- **Developer / caller**
- **Participant**
- **Video edge network**
- **Model provider**
- **Vector search service**
- **Stream Chat**

## Connections

- Developer / caller → CLI: runs commands
- Developer / caller → HTTP API: configures service
- HTTP API → Runner: dispatches requests
- Runner → Agent lifecycle: launches agent
- Participant → Video edge network: sends media
- Video edge network → Edge transport: delivers tracks
- Edge transport → Agent lifecycle: provides session
- Agent lifecycle → Inference flows: starts flow
- Inference flows → Video processors: processes video
- Inference flows → LLM interface: requests generation
- Inference flows → Speech services: transcribes and speaks
- Inference flows → Turn detection: detects turns
- Inference flows → Transcript: stores transcript
- LLM interface → Tools and MCP: dispatches tool calls
- Tools and MCP → Model provider: calls MCP services
- Inference flows → RAG retrieval: retrieves context
- RAG retrieval → Vector search service: searches vectors
- Agent lifecycle → Session registry: manages sessions
- Inference flows → Avatar sync: synchronizes media
- Avatar sync → Avatar integrations: uses avatar transport
- Phone channels → Edge transport: streams call media
- LLM interface → Model integrations: implemented by
- Video processors → Vision integrations: implemented by
- Model integrations → Model provider: sends inference
- Agent lifecycle → Metrics and events: reports metrics
- Transcript → Stream Chat: shares memory