# vrsen/openswarm architecture

> Architecture diagram of the GitHub repository vrsen/openswarm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vrsen/openswarm
- Repository: https://github.com/vrsen/openswarm

## Overview

This repo is a Python-first multi-agent system built on Agency Swarm, with a thin Node.js wrapper for distribution and a few JavaScript helpers for slide conversion. The main architecture is centered on a single orchestrator that receives a user request and delegates work to specialist agents.

The primary entrypoints are `swarm.py` for local development, `server.py` for an HTTP API runtime, and `bin/openswarm` as the packaged CLI entry. The README also shows an npm-installed launcher, so the repo supports both developer-run and end-user CLI distribution.

The orchestration boundary is explicit: `orchestrator/orchestrator.py` is the coordination layer and `orchestrator/instructions.md` defines its behavior. Its job is to route tasks, not solve them directly. This is the control plane that fans out requests to one or more specialist agents.

The specialist agents are separated into top-level modules, each with its own instructions and toolset: `virtual_assistant`, `deep_research`, `data_analyst_agent`, `slides_agent`, `docs_agent`, `image_generation_agent`, and `video_generation_agent`. Each agent is a distinct capability boundary, which makes the architecture strongly modular and easy to diagram as independent workers under one orchestrator.

A shared tool layer lives in `shared_tools/` and provides cross-cutting operations such as file copying, searching tools, finding tools, executing tools, and managing connections. This is also where model availability and OpenAI client utilities are centralized, so model/provider logic is not duplicated in every agent.

Several agents have their own internal pipelines rather than being just prompt wrappers. The slides agent is the most complex: it includes PPTX/OOXML schema assets, HTML-to-PPTX scripts, rendering utilities, slide screenshots, template registry, and slide file utilities. That indicates a two-stage flow: generate or edit slide content in HTML/PPTX-oriented intermediate forms, then export and validate to final presentation artifacts.

The docs agent follows a similar document-processing pattern. It contains tools for creating, viewing, modifying, restoring, listing, and converting documents, plus a substantial `tools/utils/` layer for HTML-to-DOCX transformation and validation. That suggests document generation is not purely textual; it includes structured document assembly and fidelity checks.

The media agents each encapsulate a specialized artifact pipeline. `image_generation_agent` includes create/edit/combine/background-removal tools plus image I/O helpers. `video_generation_agent` includes generation, trimming, audio/image/video edits, subtitles, and combine operations. These agents likely produce local media files as intermediate and final outputs rather than returning only text.

`data_analyst_agent` is built around structured-data analysis and an isolated IPython execution environment, as stated in the README. The presence of test CSV data and a separate tools package suggests a sandboxed compute boundary for analysis, plotting, and statistical work.

`virtual_assistant` is the most integration-heavy agent. Its tools cover email, calendar, Slack, filesystem operations, product search, scholar search, and skill listing. The README calls out Composio support, so this agent acts as the connector to external SaaS systems like Gmail, Slack, GitHub, and HubSpot when credentials are available.

The repo also has a clear artifact/workspace model: agents operate on files, documents, decks, images, and videos, with utilities for reading, writing, restoring, and converting those artifacts. That means the main data flow is usually prompt → orchestration → agent/tool execution → file-based deliverable, with some agents also calling external APIs for enrichment or generation.

`patches/` is an important compatibility layer. It contains targeted fixes for upstream dependencies and runtime behavior, including PPTX-related patches and file/encoding fixes. In an architecture diagram, this sits beneath the application layer as a maintenance layer that adapts third-party libraries to OpenSwarm’s workflows.

Deployment is supported in multiple ways: local Python execution, Docker via `Dockerfile` and `docker-compose.yml`, and GitHub Actions for build/test automation. The repo is therefore not just an agent library; it is a distributed delivery system for running agent workflows in terminal, container, or API form.

Main entry points: [`orchestrator/orchestrator.py`](https://github.com/vrsen/openswarm/blob/main/orchestrator/orchestrator.py) (Orchestrator), [`Dockerfile`](https://github.com/vrsen/openswarm/tree/main/Dockerfile) (Deploy), [`slides_agent/slides_agent.py`](https://github.com/vrsen/openswarm/blob/main/slides_agent/slides_agent.py) (Slides Agent), [`patches`](https://github.com/vrsen/openswarm/tree/main/patches) (Patches)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g_runtime["Runtime"]
  node_swarm_py["swarm.py<br/>local dev entrypoint<br/>[swarm.py]"]
  node_server_py["server.py<br/>http api entrypoint<br/>[server.py]"]
  node_orchestrator{{"Orchestrator<br/>routing control plane<br/>[orchestrator.py]"}}
  node_orch_instructions["Orch Rules<br/>behavior spec<br/>[instructions.md]"]
end

subgraph group_g_agents["Agents"]
  node_va_agent["Virtual Assistant<br/>external ops agent"]
  node_dr_agent["Deep Research<br/>research agent<br/>[deep_research.py]"]
  node_da_agent["Data Analyst<br/>analysis agent"]
  node_slides_agent["Slides Agent<br/>presentation pipeline<br/>[slides_agent.py]"]
  node_docs_agent["Docs Agent<br/>document pipeline<br/>[docs_agent.py]"]
  node_img_agent["Image Agent<br/>image pipeline"]
  node_vid_agent["Video Agent<br/>video pipeline"]
  node_slides_tools["Slide Tools<br/>pptx toolchain"]
  node_slides_ooxml["OOXML Assets<br/>format schemas"]
  node_docs_tools["Doc Tools<br/>document toolchain"]
  node_media_tools["Media Tools<br/>artifact tools"]
end

subgraph group_g_shared["Shared Layer"]
  node_shared_tools["Shared Tools<br/>common tool layer"]
  node_patches["Patches<br/>compatibility layer"]
end

subgraph group_g_delivery["Delivery"]
  node_cli_bin["openswarm<br/>packaged cli"]
  node_deploy["Deploy<br/>container and ci"]
end

node_swarm_py -->|"starts"| node_orchestrator
node_server_py -->|"serves"| node_orchestrator
node_cli_bin -->|"launches"| node_orchestrator
node_orchestrator -->|"guided by"| node_orch_instructions
node_orchestrator -->|"delegates"| node_va_agent
node_orchestrator -->|"delegates"| node_dr_agent
node_orchestrator -->|"delegates"| node_da_agent
node_orchestrator -->|"delegates"| node_slides_agent
node_orchestrator -->|"delegates"| node_docs_agent
node_orchestrator -->|"delegates"| node_img_agent
node_orchestrator -->|"delegates"| node_vid_agent
node_va_agent -->|"uses"| node_shared_tools
node_da_agent -->|"uses"| node_shared_tools
node_slides_agent -->|"uses"| node_slides_tools
node_slides_agent -->|"uses"| node_slides_ooxml
node_slides_tools -.->|"adapted by"| node_patches
node_docs_agent -->|"uses"| node_docs_tools
node_docs_tools -.->|"adapted by"| node_patches
node_img_agent -->|"uses"| node_media_tools
node_vid_agent -->|"uses"| node_media_tools
node_patches -.->|"stabilizes"| node_shared_tools
node_deploy -->|"packages"| node_swarm_py
node_deploy -->|"packages"| node_server_py
node_deploy -->|"packages"| node_cli_bin

click node_swarm_py "https://github.com/vrsen/openswarm/blob/main/swarm.py"
click node_server_py "https://github.com/vrsen/openswarm/blob/main/server.py"
click node_cli_bin "https://github.com/vrsen/openswarm/tree/main/bin/openswarm"
click node_orchestrator "https://github.com/vrsen/openswarm/blob/main/orchestrator/orchestrator.py"
click node_orch_instructions "https://github.com/vrsen/openswarm/blob/main/orchestrator/instructions.md"
click node_va_agent "https://github.com/vrsen/openswarm/blob/main/virtual_assistant/virtual_assistant.py"
click node_dr_agent "https://github.com/vrsen/openswarm/blob/main/deep_research/deep_research.py"
click node_da_agent "https://github.com/vrsen/openswarm/blob/main/data_analyst_agent/data_analyst_agent.py"
click node_slides_agent "https://github.com/vrsen/openswarm/blob/main/slides_agent/slides_agent.py"
click node_docs_agent "https://github.com/vrsen/openswarm/blob/main/docs_agent/docs_agent.py"
click node_img_agent "https://github.com/vrsen/openswarm/blob/main/image_generation_agent/image_generation_agent.py"
click node_vid_agent "https://github.com/vrsen/openswarm/blob/main/video_generation_agent/video_generation_agent.py"
click node_shared_tools "https://github.com/vrsen/openswarm/tree/main/shared_tools"
click node_patches "https://github.com/vrsen/openswarm/tree/main/patches"
click node_slides_tools "https://github.com/vrsen/openswarm/tree/main/slides_agent/tools"
click node_slides_ooxml "https://github.com/vrsen/openswarm/tree/main/slides_agent/pptx/ooxml"
click node_docs_tools "https://github.com/vrsen/openswarm/tree/main/docs_agent/tools"
click node_media_tools "https://github.com/vrsen/openswarm/tree/main/image_generation_agent/tools"
click node_deploy "https://github.com/vrsen/openswarm/tree/main/Dockerfile"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_swarm_py,node_server_py,node_orchestrator,node_orch_instructions toneBlue
class node_va_agent,node_dr_agent,node_da_agent,node_slides_agent,node_docs_agent,node_img_agent,node_vid_agent,node_slides_tools,node_slides_ooxml,node_docs_tools,node_media_tools toneAmber
class node_shared_tools,node_patches toneMint
class node_cli_bin,node_deploy toneRose
```

## Components

### Runtime

Entry points and orchestration control plane

- [swarm.py](https://github.com/vrsen/openswarm/blob/main/swarm.py): local dev entrypoint · Python entrypoint for local agent runs · `swarm.py`
- [server.py](https://github.com/vrsen/openswarm/blob/main/server.py): http api entrypoint · Runs the system behind an HTTP API · `server.py`
- [Orchestrator](https://github.com/vrsen/openswarm/blob/main/orchestrator/orchestrator.py): routing control plane · Receives requests and delegates work to specialist agents · `orchestrator/orchestrator.py`
- [Orch Rules](https://github.com/vrsen/openswarm/blob/main/orchestrator/instructions.md): behavior spec · Instruction set defining orchestration behavior · `orchestrator/instructions.md`

### Agents

Specialist workers and their tool pipelines

- [Virtual Assistant](https://github.com/vrsen/openswarm/blob/main/virtual_assistant/virtual_assistant.py): external ops agent · Handles email, calendar, Slack, filesystem, search, and skills · `virtual_assistant/virtual_assistant.py`
- [Deep Research](https://github.com/vrsen/openswarm/blob/main/deep_research/deep_research.py): research agent · Research-oriented specialist agent · `deep_research/deep_research.py`
- [Data Analyst](https://github.com/vrsen/openswarm/blob/main/data_analyst_agent/data_analyst_agent.py): analysis agent · Structured-data analysis and sandboxed compute agent · `data_analyst_agent/data_analyst_agent.py`
- [Slides Agent](https://github.com/vrsen/openswarm/blob/main/slides_agent/slides_agent.py): presentation pipeline · Builds and edits slide decks through HTML and PPTX stages · `slides_agent/slides_agent.py`
- [Docs Agent](https://github.com/vrsen/openswarm/blob/main/docs_agent/docs_agent.py): document pipeline · Creates, modifies, and converts documents with validation · `docs_agent/docs_agent.py`
- [Image Agent](https://github.com/vrsen/openswarm/blob/main/image_generation_agent/image_generation_agent.py): image pipeline · Generates and edits images, including background removal · `image_generation_agent/image_generation_agent.py`
- [Video Agent](https://github.com/vrsen/openswarm/blob/main/video_generation_agent/video_generation_agent.py): video pipeline · Generates and edits video, audio, subtitles, and composites · `video_generation_agent/video_generation_agent.py`
- [Slide Tools](https://github.com/vrsen/openswarm/tree/main/slides_agent/tools): pptx toolchain · Slide build, render, inspect, and template utilities · `slides_agent/tools`
- [OOXML Assets](https://github.com/vrsen/openswarm/tree/main/slides_agent/pptx/ooxml): format schemas · PPTX/OOXML schemas, pack/unpack, and validation scripts · `slides_agent/pptx/ooxml`
- [Doc Tools](https://github.com/vrsen/openswarm/tree/main/docs_agent/tools): document toolchain · Document creation, conversion, restore, and HTML-to-DOCX helpers · `docs_agent/tools`
- [Media Tools](https://github.com/vrsen/openswarm/tree/main/image_generation_agent/tools): artifact tools · Image and video generation/editing utilities and shared helpers · `image_generation_agent/tools`

### Shared Layer

Cross-cutting tools, patches, and common utilities

- [Shared Tools](https://github.com/vrsen/openswarm/tree/main/shared_tools): common tool layer · Copying, searching, executing, connection management, and model utilities · `shared_tools`
- [Patches](https://github.com/vrsen/openswarm/tree/main/patches): compatibility layer · Targeted fixes for upstream libraries and runtime quirks · `patches`

### Delivery

Packaging and deployment surfaces

- [openswarm](https://github.com/vrsen/openswarm/tree/main/bin/openswarm): packaged cli · Packaged command-line launcher · `bin/openswarm`
- [Deploy](https://github.com/vrsen/openswarm/tree/main/Dockerfile): container and ci · Docker and GitHub Actions delivery surfaces · `Dockerfile`

## Connections

- swarm.py → Orchestrator: starts · Local development runtime enters the control plane
- server.py → Orchestrator: serves · HTTP runtime delegates requests into orchestration
- openswarm → Orchestrator: launches · Packaged CLI invokes the same orchestration layer
- Orchestrator → Orch Rules: guided by · Routing behavior is defined by orchestration instructions
- Orchestrator → Virtual Assistant: delegates · Dispatches tasks to the virtual assistant specialist
- Orchestrator → Deep Research: delegates · Dispatches research work to the research agent
- Orchestrator → Data Analyst: delegates · Dispatches analysis tasks to the analyst agent
- Orchestrator → Slides Agent: delegates · Dispatches deck work to the slides pipeline
- Orchestrator → Docs Agent: delegates · Dispatches document work to the docs pipeline
- Orchestrator → Image Agent: delegates · Dispatches image tasks to the image pipeline
- Orchestrator → Video Agent: delegates · Dispatches video tasks to the video pipeline
- Virtual Assistant → Shared Tools: uses · Shares connection, search, and execution utilities
- Data Analyst → Shared Tools: uses · Relies on shared model/client helpers and execution support
- Slides Agent → Slide Tools: uses · Presentation assembly depends on dedicated slide tooling
- Slides Agent → OOXML Assets: uses · Uses schema assets and pack/unpack validation tooling
- Slide Tools → Patches: adapted by · Slide workflow depends on compatibility fixes
- Docs Agent → Doc Tools: uses · Document lifecycle actions run through the docs toolchain
- Doc Tools → Patches: adapted by · HTML-to-DOCX and file handling are stabilized by patches
- Image Agent → Media Tools: uses · Image generation and editing use media helpers
- Video Agent → Media Tools: uses · Video generation and editing use media helpers
- Patches → Shared Tools: stabilizes · Compatibility fixes support shared runtime behavior
- Deploy → swarm.py: packages · Container and CI delivery include the Python runtime
- Deploy → server.py: packages · Container and CI delivery include the API runtime
- Deploy → openswarm: packages · Release artifacts include the CLI launcher