How understudy-ai/understudy works
1. Repository shape
- This is a TypeScript monorepo organized around one local agent platform.
- The main runtime pieces are split into apps/cli, packages/gateway, packages/core, packages/tools, packages/gui, packages/channels, packages/plugins, and packages/types.
- Built-in skills live in skills/, with examples and published skill artifacts under examples/.
2. Top-level control plane
- The system is centered on a single gateway process that exposes HTTP, WebSocket, and JSON-RPC surfaces.
- User-facing entrypoints include the terminal CLI, dashboard, web chat, and external messaging channels.
- These all converge on the gateway rather than each talking directly to tools or models.
3. CLI as operator front door
- apps/cli is the user’s primary terminal interface and also the process launcher for daemon/gateway mode.
- It contains commands for chat, agent execution, setup/wizard flows, browser relay setup, model and skill inspection, channels, sessions, health, and teaching.
- The CLI uses an RPC client to reach the running gateway when not operating as the gateway itself.
4. Gateway as orchestration layer
- packages/gateway is the application server and orchestration boundary.
- It contains routing, handler registration, session runtime management, playbook runtime, worker runtime, event bus, security/auth, rate limiting, and UI helpers.
- This is where inbound messages are turned into session work, where responses are emitted back to clients, and where multi-stage pipelines are coordinated.
5. Core runtime and policy engine
- packages/core holds the reusable agent runtime primitives shared across entrypoints.
- Key responsibilities include config/auth, system-prompt assembly, tool registry, policy pipeline, trust engine, usage tracking, session traces, workspace artifacts, runtime adapters, and model-resolution bridging.
- It defines the guardrails and execution policies that every tool call and agent step must pass through.
6. Tool execution layer
- packages/tools is the capability layer invoked by sessions.
- It includes browser automation, GUI-related tools, shell/process execution, web search and fetch, memory, scheduling, messaging, image/PDF/video helpers, runtime status, and bridge tools for subagents and gateway integration.
- The agent runtime selects among these tools based on the task and policy constraints.
7. Native GUI subsystem
- packages/gui is the macOS-native execution backend for human-like computer control.
- It provides screen capture, input injection, screenshot-grounding support, readiness checks, physical resource locking, and demonstration recording.
- This subsystem depends on macOS permissions and a compiled native helper, so it is the boundary for all desktop automation and teach-by-demonstration recording.
8. Browser and external app attachment
- Browser control is split between managed browser automation and Chrome extension relay support.
- The extension assets under assets/chrome-extension/ allow the runtime to attach to a real logged-in Chrome session.
- This gives the agent two browser routes: a clean managed browser and an attached real-browser path.
9. Messaging and channel adapters
- packages/channels contains adapters for Web, Telegram, Discord, Slack, WhatsApp, Signal, LINE, and iMessage.
- These channels are a separate boundary from the core agent loop: they translate platform-specific message formats into the shared channel/message model.
- Incoming channel messages become gateway sessions; outgoing results are routed back through the same adapter.
10. Skills, teach, and workspace artifacts
- skills/ contains built-in reusable skills, while examples/ shows published skill artifacts.
- The repo models three artifact kinds: skills, workers, and playbooks.
- Teach-by-demonstration and workflow crystallization both produce workspace artifacts that the gateway can hot-load and reuse in later sessions.
11. Data and execution flow
- A typical request enters from CLI, dashboard, web chat, or a messaging channel.
- The gateway creates or resumes a session, applies the policy pipeline from core, and invokes tools from packages/tools or subagents/playbooks as needed.
- Output, traces, recordings, and workspace artifacts are stored locally by default, with model-provider calls used only where the runtime needs reasoning or grounding.
12. Extension and packaging boundaries
- packages/plugins provides a loader/registry pattern for dynamically discovered plugin modules.
- packages/types defines shared contracts so CLI, gateway, tools, and channels agree on config, channel, and tool-schema shapes.
- The monorepo is built and tested with pnpm workspaces, with GitHub Actions for CI, publishing, and docs deployment.
Main entry points: server.ts Gateway · session-runtime.ts Sessions · src/index.ts Core · src/index.ts Tools
Components 20
Entrypoints
User-facing launchers and surfaces
- CLI · Primary operator front door and daemon launcher
apps/cli/src/index.ts - RPC client · CLI client used to talk to a running gateway
apps/cli/src/rpc-client.ts - Commands · Chat, setup, daemon, sessions, health, skills, browser, and teaching commands
apps/cli/src/commands
Gateway
Central session and request orchestration
- Gateway · Central HTTP, WebSocket, and JSON-RPC orchestration boundary
packages/gateway/src/server.ts - Router · Routes inbound requests to the right session and handler path
packages/gateway/src/router.ts - Sessions · Creates, resumes, and coordinates session work
packages/gateway/src/session-runtime.ts - Playbooks · Runs structured multi-stage workflows and crystallized work
packages/gateway/src/playbook-runtime.ts - Workers · Executes worker-backed task steps and subflows
packages/gateway/src/worker-runtime.ts - Handlers · Chat, session, channel, discovery, config, schedule, and usage handlers
packages/gateway/src/handlers
Core runtime
Shared agent policy and execution primitives
- Core · Reusable runtime primitives, policy, prompt assembly, and adapters
packages/core/src/index.ts - Policy · Guardrails, trust, routing, tool sanitization, and reply policies
packages/core/src/runtime/policy-pipeline.ts - Tool registry · Shared registration and selection of tool capabilities
packages/core/src/tool-registry.ts - Artifacts · Skills, traces, drafts, and workflow-crystallized outputs
packages/core/src/workspace-artifacts.ts
Capabilities
Tools and desktop/browser execution backends
- Tools · Browser, shell, memory, schedule, media, search, and bridge tools
packages/tools/src/index.ts - Browser · Managed browser control and extension relay support
packages/tools/src/browser - GUI backend · Native screen capture, input injection, grounding, and recording
packages/gui/src/runtime.ts
Integrations
Messaging and plugin boundaries
Connections 21
- CLI → Commands: invokes · CLI entrypoint dispatches to commands
- CLI → RPC client: uses · Non-daemon CLI talks to gateway over RPC
- Commands → Gateway: launches / queries · Daemon, health, session, and browser commands reach gateway services
- RPC client → Gateway: connects to · RPC client targets the running gateway process
- Gateway → Router: routes through · Inbound protocol traffic is routed to handlers and sessions
- Router → Handlers: dispatches · Request and event routing selects the appropriate handler set
- Handlers → Sessions: opens work · Chat and session handlers create or resume sessions
- Sessions → Core: depends on · Session execution relies on shared runtime primitives
- Core → Policy: applies · Policy pipeline guards prompts, tools, and replies
- Core → Tool registry: selects · Runtime resolves capabilities through the tool registry
- Core → Artifacts: writes · Runtime persists traces, drafts, and workspace artifacts
- Sessions → Playbooks: coordinates · Sessions can run structured playbooks
- Sessions → Workers: spawns · Sessions can delegate to worker runtimes
- Sessions → Tools: invokes · Session work executes through tools
- Tools → Browser: contains · Browser automation and relay paths live in the tools layer
- Tools → GUI backend: uses · Desktop automation tools rely on the macOS GUI backend
- Handlers → Channels: adapts · Channel handlers bridge external message platforms into sessions
- Gateway → Plugins: loads · Gateway can discover and register plugin modules
- Gateway → Skills: hot-loads · Gateway reuses built-in skills during sessions and teaching
- Gateway → Examples: consumes · Examples document and package published skill artifacts
- Tools → Gateway: bridges to · Bridge tools let sessions reach gateway services and subagents