# understudy-ai/understudy architecture

> Architecture diagram of the GitHub repository understudy-ai/understudy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/understudy-ai/understudy
- Repository: https://github.com/understudy-ai/understudy

## Overview

1. **Repository shape**
   - This is a TypeScript monorepo organized around one local agent platform.
   - The main runtime pieces are split into `apps/cli`, `packages/gateway`, `packages/core`, `packages/tools`, `packages/gui`, `packages/channels`, `packages/plugins`, and `packages/types`.
   - Built-in skills live in `skills/`, with examples and published skill artifacts under `examples/`.

2. **Top-level control plane**
   - The system is centered on a single **gateway** process that exposes HTTP, WebSocket, and JSON-RPC surfaces.
   - User-facing entrypoints include the terminal CLI, dashboard, web chat, and external messaging channels.
   - These all converge on the gateway rather than each talking directly to tools or models.

3. **CLI as operator front door**
   - `apps/cli` is the user’s primary terminal interface and also the process launcher for daemon/gateway mode.
   - It contains commands for chat, agent execution, setup/wizard flows, browser relay setup, model and skill inspection, channels, sessions, health, and teaching.
   - The CLI uses an RPC client to reach the running gateway when not operating as the gateway itself.

4. **Gateway as orchestration layer**
   - `packages/gateway` is the application server and orchestration boundary.
   - It contains routing, handler registration, session runtime management, playbook runtime, worker runtime, event bus, security/auth, rate limiting, and UI helpers.
   - This is where inbound messages are turned into session work, where responses are emitted back to clients, and where multi-stage pipelines are coordinated.

5. **Core runtime and policy engine**
   - `packages/core` holds the reusable agent runtime primitives shared across entrypoints.
   - Key responsibilities include config/auth, system-prompt assembly, tool registry, policy pipeline, trust engine, usage tracking, session traces, workspace artifacts, runtime adapters, and model-resolution bridging.
   - It defines the guardrails and execution policies that every tool call and agent step must pass through.

6. **Tool execution layer**
   - `packages/tools` is the capability layer invoked by sessions.
   - It includes browser automation, GUI-related tools, shell/process execution, web search and fetch, memory, scheduling, messaging, image/PDF/video helpers, runtime status, and bridge tools for subagents and gateway integration.
   - The agent runtime selects among these tools based on the task and policy constraints.

7. **Native GUI subsystem**
   - `packages/gui` is the macOS-native execution backend for human-like computer control.
   - It provides screen capture, input injection, screenshot-grounding support, readiness checks, physical resource locking, and demonstration recording.
   - This subsystem depends on macOS permissions and a compiled native helper, so it is the boundary for all desktop automation and teach-by-demonstration recording.

8. **Browser and external app attachment**
   - Browser control is split between managed browser automation and Chrome extension relay support.
   - The extension assets under `assets/chrome-extension/` allow the runtime to attach to a real logged-in Chrome session.
   - This gives the agent two browser routes: a clean managed browser and an attached real-browser path.

9. **Messaging and channel adapters**
   - `packages/channels` contains adapters for Web, Telegram, Discord, Slack, WhatsApp, Signal, LINE, and iMessage.
   - These channels are a separate boundary from the core agent loop: they translate platform-specific message formats into the shared channel/message model.
   - Incoming channel messages become gateway sessions; outgoing results are routed back through the same adapter.

10. **Skills, teach, and workspace artifacts**
   - `skills/` contains built-in reusable skills, while `examples/` shows published skill artifacts.
   - The repo models three artifact kinds: **skills**, **workers**, and **playbooks**.
   - Teach-by-demonstration and workflow crystallization both produce workspace artifacts that the gateway can hot-load and reuse in later sessions.

11. **Data and execution flow**
   - A typical request enters from CLI, dashboard, web chat, or a messaging channel.
   - The gateway creates or resumes a session, applies the policy pipeline from `core`, and invokes tools from `packages/tools` or subagents/playbooks as needed.
   - Output, traces, recordings, and workspace artifacts are stored locally by default, with model-provider calls used only where the runtime needs reasoning or grounding.

12. **Extension and packaging boundaries**
   - `packages/plugins` provides a loader/registry pattern for dynamically discovered plugin modules.
   - `packages/types` defines shared contracts so CLI, gateway, tools, and channels agree on config, channel, and tool-schema shapes.
   - The monorepo is built and tested with pnpm workspaces, with GitHub Actions for CI, publishing, and docs deployment.

Main entry points: [`packages/gateway/src/server.ts`](https://github.com/understudy-ai/understudy/blob/main/packages/gateway/src/server.ts) (Gateway), [`packages/gateway/src/session-runtime.ts`](https://github.com/understudy-ai/understudy/blob/main/packages/gateway/src/session-runtime.ts) (Sessions), [`packages/core/src/index.ts`](https://github.com/understudy-ai/understudy/blob/main/packages/core/src/index.ts) (Core), [`packages/tools/src/index.ts`](https://github.com/understudy-ai/understudy/blob/main/packages/tools/src/index.ts) (Tools)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entrypoints["Entrypoints"]
  node_apps_cli["CLI<br/>terminal app<br/>[index.ts]"]
  node_cli_rpc["RPC client<br/>gateway client<br/>[rpc-client.ts]"]
  node_cli_commands["Commands<br/>cli command set"]
end

subgraph group_orchestration["Gateway"]
  node_gateway_server["Gateway<br/>app server<br/>[server.ts]"]
  node_gateway_router["Router<br/>request routing<br/>[router.ts]"]
  node_gateway_sessions["Sessions<br/>session runtime<br/>[session-runtime.ts]"]
  node_gateway_playbooks["Playbooks<br/>workflow runtime"]
  node_gateway_workers["Workers<br/>worker runtime<br/>[worker-runtime.ts]"]
  node_gateway_handlers["Handlers"]
end

subgraph group_runtime["Core runtime"]
  node_core_runtime["Core<br/>agent runtime<br/>[index.ts]"]
  node_core_policy["Policy<br/>[policy-pipeline.ts]"]
  node_core_tools["Tool registry<br/>[tool-registry.ts]"]
  node_core_artifacts["Artifacts"]
end

subgraph group_capabilities["Capabilities"]
  node_tools_suite["Tools<br/>capability layer<br/>[index.ts]"]
  node_browser_tools["Browser"]
  node_gui_backend["GUI backend<br/>macOS control<br/>[runtime.ts]"]
end

subgraph group_integrations["Integrations"]
  node_channels_pkg["Channels<br/>message adapters<br/>[index.ts]"]
  node_plugins_pkg["Plugins<br/>plugin loader<br/>[index.ts]"]
end

subgraph group_artifacts["Artifacts"]
  node_skills_dir["Skills<br/>skill library"]
  node_examples_dir["Examples<br/>published artifacts"]
end

node_apps_cli -->|"invokes"| node_cli_commands
node_apps_cli -->|"uses"| node_cli_rpc
node_cli_commands -->|"launches / queries"| node_gateway_server
node_cli_rpc -->|"connects to"| node_gateway_server
node_gateway_server -->|"routes through"| node_gateway_router
node_gateway_router -->|"dispatches"| node_gateway_handlers
node_gateway_handlers -->|"opens work"| node_gateway_sessions
node_gateway_sessions -->|"depends on"| node_core_runtime
node_core_runtime -->|"applies"| node_core_policy
node_core_runtime -->|"selects"| node_core_tools
node_core_runtime -->|"writes"| node_core_artifacts
node_gateway_sessions -->|"coordinates"| node_gateway_playbooks
node_gateway_sessions -->|"spawns"| node_gateway_workers
node_gateway_sessions -->|"invokes"| node_tools_suite
node_tools_suite -->|"contains"| node_browser_tools
node_tools_suite -->|"uses"| node_gui_backend
node_gateway_handlers -->|"adapts"| node_channels_pkg
node_gateway_server -->|"loads"| node_plugins_pkg
node_gateway_server -->|"hot-loads"| node_skills_dir
node_gateway_server -->|"consumes"| node_examples_dir
node_tools_suite -->|"bridges to"| node_gateway_server

click node_apps_cli "https://github.com/understudy-ai/understudy/blob/main/apps/cli/src/index.ts"
click node_cli_rpc "https://github.com/understudy-ai/understudy/blob/main/apps/cli/src/rpc-client.ts"
click node_cli_commands "https://github.com/understudy-ai/understudy/tree/main/apps/cli/src/commands"
click node_gateway_server "https://github.com/understudy-ai/understudy/blob/main/packages/gateway/src/server.ts"
click node_gateway_router "https://github.com/understudy-ai/understudy/blob/main/packages/gateway/src/router.ts"
click node_gateway_sessions "https://github.com/understudy-ai/understudy/blob/main/packages/gateway/src/session-runtime.ts"
click node_gateway_playbooks "https://github.com/understudy-ai/understudy/blob/main/packages/gateway/src/playbook-runtime.ts"
click node_gateway_workers "https://github.com/understudy-ai/understudy/blob/main/packages/gateway/src/worker-runtime.ts"
click node_gateway_handlers "https://github.com/understudy-ai/understudy/tree/main/packages/gateway/src/handlers"
click node_core_runtime "https://github.com/understudy-ai/understudy/blob/main/packages/core/src/index.ts"
click node_core_policy "https://github.com/understudy-ai/understudy/blob/main/packages/core/src/runtime/policy-pipeline.ts"
click node_core_tools "https://github.com/understudy-ai/understudy/blob/main/packages/core/src/tool-registry.ts"
click node_core_artifacts "https://github.com/understudy-ai/understudy/blob/main/packages/core/src/workspace-artifacts.ts"
click node_tools_suite "https://github.com/understudy-ai/understudy/blob/main/packages/tools/src/index.ts"
click node_browser_tools "https://github.com/understudy-ai/understudy/tree/main/packages/tools/src/browser"
click node_gui_backend "https://github.com/understudy-ai/understudy/blob/main/packages/gui/src/runtime.ts"
click node_channels_pkg "https://github.com/understudy-ai/understudy/blob/main/packages/channels/src/index.ts"
click node_plugins_pkg "https://github.com/understudy-ai/understudy/blob/main/packages/plugins/src/index.ts"
click node_skills_dir "https://github.com/understudy-ai/understudy/tree/main/skills"
click node_examples_dir "https://github.com/understudy-ai/understudy/tree/main/examples"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_apps_cli,node_cli_rpc,node_cli_commands toneBlue
class node_gateway_server,node_gateway_router,node_gateway_sessions,node_gateway_playbooks,node_gateway_workers,node_gateway_handlers toneAmber
class node_core_runtime,node_core_policy,node_core_tools,node_core_artifacts toneMint
class node_tools_suite,node_browser_tools,node_gui_backend toneRose
class node_channels_pkg,node_plugins_pkg toneIndigo
class node_skills_dir,node_examples_dir toneTeal
```

## Components

### Entrypoints

User-facing launchers and surfaces

- [CLI](https://github.com/understudy-ai/understudy/blob/main/apps/cli/src/index.ts): terminal app · Primary operator front door and daemon launcher · `apps/cli/src/index.ts`
- [RPC client](https://github.com/understudy-ai/understudy/blob/main/apps/cli/src/rpc-client.ts): gateway client · CLI client used to talk to a running gateway · `apps/cli/src/rpc-client.ts`
- [Commands](https://github.com/understudy-ai/understudy/tree/main/apps/cli/src/commands): cli command set · Chat, setup, daemon, sessions, health, skills, browser, and teaching commands · `apps/cli/src/commands`

### Gateway

Central session and request orchestration

- [Gateway](https://github.com/understudy-ai/understudy/blob/main/packages/gateway/src/server.ts): app server · Central HTTP, WebSocket, and JSON-RPC orchestration boundary · `packages/gateway/src/server.ts`
- [Router](https://github.com/understudy-ai/understudy/blob/main/packages/gateway/src/router.ts): request routing · Routes inbound requests to the right session and handler path · `packages/gateway/src/router.ts`
- [Sessions](https://github.com/understudy-ai/understudy/blob/main/packages/gateway/src/session-runtime.ts): session runtime · Creates, resumes, and coordinates session work · `packages/gateway/src/session-runtime.ts`
- [Playbooks](https://github.com/understudy-ai/understudy/blob/main/packages/gateway/src/playbook-runtime.ts): workflow runtime · Runs structured multi-stage workflows and crystallized work · `packages/gateway/src/playbook-runtime.ts`
- [Workers](https://github.com/understudy-ai/understudy/blob/main/packages/gateway/src/worker-runtime.ts): worker runtime · Executes worker-backed task steps and subflows · `packages/gateway/src/worker-runtime.ts`
- [Handlers](https://github.com/understudy-ai/understudy/tree/main/packages/gateway/src/handlers): request handlers · Chat, session, channel, discovery, config, schedule, and usage handlers · `packages/gateway/src/handlers`

### Core runtime

Shared agent policy and execution primitives

- [Core](https://github.com/understudy-ai/understudy/blob/main/packages/core/src/index.ts): agent runtime · Reusable runtime primitives, policy, prompt assembly, and adapters · `packages/core/src/index.ts`
- [Policy](https://github.com/understudy-ai/understudy/blob/main/packages/core/src/runtime/policy-pipeline.ts): policy pipeline · Guardrails, trust, routing, tool sanitization, and reply policies · `packages/core/src/runtime/policy-pipeline.ts`
- [Tool registry](https://github.com/understudy-ai/understudy/blob/main/packages/core/src/tool-registry.ts): tool registry · Shared registration and selection of tool capabilities · `packages/core/src/tool-registry.ts`
- [Artifacts](https://github.com/understudy-ai/understudy/blob/main/packages/core/src/workspace-artifacts.ts): workspace artifacts · Skills, traces, drafts, and workflow-crystallized outputs · `packages/core/src/workspace-artifacts.ts`

### Capabilities

Tools and desktop/browser execution backends

- [Tools](https://github.com/understudy-ai/understudy/blob/main/packages/tools/src/index.ts): capability layer · Browser, shell, memory, schedule, media, search, and bridge tools · `packages/tools/src/index.ts`
- [Browser](https://github.com/understudy-ai/understudy/tree/main/packages/tools/src/browser): browser automation · Managed browser control and extension relay support · `packages/tools/src/browser`
- [GUI backend](https://github.com/understudy-ai/understudy/blob/main/packages/gui/src/runtime.ts): macOS control · Native screen capture, input injection, grounding, and recording · `packages/gui/src/runtime.ts`

### Integrations

Messaging and plugin boundaries

- [Channels](https://github.com/understudy-ai/understudy/blob/main/packages/channels/src/index.ts): message adapters · Web, Telegram, Discord, Slack, WhatsApp, Signal, LINE, and iMessage adapters · `packages/channels/src/index.ts`
- [Plugins](https://github.com/understudy-ai/understudy/blob/main/packages/plugins/src/index.ts): plugin loader · Dynamic plugin discovery and registry · `packages/plugins/src/index.ts`

### Artifacts

Skills and published workflow outputs

- [Skills](https://github.com/understudy-ai/understudy/tree/main/skills): skill library · Built-in reusable skills and teachable workflows · `skills`
- [Examples](https://github.com/understudy-ai/understudy/tree/main/examples): published artifacts · Example and published skill artifacts · `examples`

## Connections

- CLI → Commands: invokes · CLI entrypoint dispatches to commands
- CLI → RPC client: uses · Non-daemon CLI talks to gateway over RPC
- Commands → Gateway: launches / queries · Daemon, health, session, and browser commands reach gateway services
- RPC client → Gateway: connects to · RPC client targets the running gateway process
- Gateway → Router: routes through · Inbound protocol traffic is routed to handlers and sessions
- Router → Handlers: dispatches · Request and event routing selects the appropriate handler set
- Handlers → Sessions: opens work · Chat and session handlers create or resume sessions
- Sessions → Core: depends on · Session execution relies on shared runtime primitives
- Core → Policy: applies · Policy pipeline guards prompts, tools, and replies
- Core → Tool registry: selects · Runtime resolves capabilities through the tool registry
- Core → Artifacts: writes · Runtime persists traces, drafts, and workspace artifacts
- Sessions → Playbooks: coordinates · Sessions can run structured playbooks
- Sessions → Workers: spawns · Sessions can delegate to worker runtimes
- Sessions → Tools: invokes · Session work executes through tools
- Tools → Browser: contains · Browser automation and relay paths live in the tools layer
- Tools → GUI backend: uses · Desktop automation tools rely on the macOS GUI backend
- Handlers → Channels: adapts · Channel handlers bridge external message platforms into sessions
- Gateway → Plugins: loads · Gateway can discover and register plugin modules
- Gateway → Skills: hot-loads · Gateway reuses built-in skills during sessions and teaching
- Gateway → Examples: consumes · Examples document and package published skill artifacts
- Tools → Gateway: bridges to · Bridge tools let sessions reach gateway services and subagents
