# safiyev/heyagent architecture

> Architecture diagram of the GitHub repository safiyev/heyagent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/safiyev/heyagent
- Repository: https://github.com/safiyev/heyagent

## Overview

#### Purpose and execution surfaces
HeyAgent is a local-first TypeScript/Node.js desktop automation agent. It accepts tasks through the terminal, Electron companion, or a locally hosted gateway with Telegram support. Primary entry points are `apps/cli/src/cli.ts`, `apps/gateway/src/index.ts`, and `apps/desktop/src/main.ts`; shared agent runtime starts at `packages/agent/src/index.ts`.

#### Interface adapters and local gateway
The CLI handles onboarding, configuration, one-shot tasks, and chat sessions. The Electron app is a companion UI that talks to the loopback gateway. The gateway is the long-running boundary for local API access, Telegram polling, scheduled jobs, and background missions; it should remain bound to localhost. Relevant paths: `apps/cli/src/index.ts`, `apps/desktop/src/ui/app.js`, `apps/gateway/src/index.ts`.

#### Agent context, sessions, and skills
The agent package owns conversation/session management, task normalization, identity-backed memory, mission history, queued sessions, chat commands, and dynamically loaded skill instructions. Its runtime converts user text into normalized work with persistent local state under `~/.heyagent`. Relevant paths: `packages/agent/src/session.ts`, `packages/agent/src/memory.ts`, `packages/agent/src/skills-loader.ts`.

#### Deterministic orchestration pipeline
The orchestrator implements the core control flow: route an intent, create a task contract and plan graph, enqueue execution, run steps, record actions/world state, then verify criteria. Retry, recovery, waits, idempotency, and timeline persistence make mission execution resilient rather than a single model call. Relevant paths: `packages/orchestrator/src/orchestrator.ts`, `packages/orchestrator/src/planner.ts`, `packages/orchestrator/src/step-runner.ts`.

#### Harness/tool-loop execution boundary
Plans are dispatched either to deterministic harnesses for recognized workflows or to an LLM-driven tool loop. The agent supplies tool definitions for desktop, integrations, memory, scheduling, screen, and power operations; loop protections and compaction manage long-running model interactions. Relevant paths: `packages/agent/src/harness/execute.ts`, `packages/agent/src/tool-loop.ts`, `packages/agent/src/tools.ts`.

#### Desktop and browser control
`packages/computer` is the privileged local-machine adapter: screenshots/vision, mouse and keyboard input, windows/accessibility, browser navigation/profile access, files/shell-adjacent operations, media, and platform system controls. Windows has the richest implementation, with cross-platform alternatives where possible. Relevant paths: `packages/computer/src/index.ts`, `packages/computer/src/browser-agent.ts`, `packages/computer/src/system-controls.ts`.

#### Models, integrations, policy, and external services
The model layer selects providers, validates responses, manages credentials, and fails over among cloud, Bedrock/OpenRouter-compatible, or Ollama models. Integrations connect Google OAuth Workspace APIs, Gmail IMAP, Notion, and GitHub. Policy gates capabilities and transaction approvals before sensitive execution. Telegram transport and optional OpenAI Whisper STT/ElevenLabs TTS are external channels/services. Relevant paths: `packages/models/src/providers.ts`, `packages/integrations/src/google-oauth.ts`, `packages/policy/src/transaction-approvals.ts`, `packages/channels-telegram/src/index.ts`.

Main entry points: [`packages/orchestrator/src/step-runner.ts`](https://github.com/safiyev/heyagent/blob/main/packages/orchestrator/src/step-runner.ts) (Step runner), [`packages/agent/src/index.ts`](https://github.com/safiyev/heyagent/blob/main/packages/agent/src/index.ts) (Agent runtime), [`packages/agent/src/tools.ts`](https://github.com/safiyev/heyagent/blob/main/packages/agent/src/tools.ts) (Agent tools), [`packages/orchestrator/src/orchestrator.ts`](https://github.com/safiyev/heyagent/blob/main/packages/orchestrator/src/orchestrator.ts) (Orchestrator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["Local interfaces"]
  node_cli["CLI<br/>terminal adapter<br/>[cli.ts]"]
  node_desktop["Electron companion<br/>desktop UI<br/>[main.ts]"]
  node_gateway["Local gateway<br/>localhost service<br/>[index.ts]"]
  node_telegram["Telegram transport<br/>channel adapter<br/>[index.ts]"]
end

subgraph group_core["Agent runtime"]
  node_agent["Agent runtime<br/>runtime facade<br/>[index.ts]"]
  node_sessions["Sessions and memory<br/>state manager<br/>[session.ts]"]
  node_local_state[("Local agent state<br/>persistent state")]
  node_skills["Skill loader<br/>instruction loader<br/>[skills-loader.ts]"]
end

subgraph group_execution["Execution"]
  node_orchestrator["Orchestrator<br/>mission controller<br/>[orchestrator.ts]"]
  node_planner["Task planner<br/>plan graph builder<br/>[planner.ts]"]
  node_step_runner["Step runner<br/>queued executor<br/>[step-runner.ts]"]
  node_harness["Deterministic harness<br/>workflow executor<br/>[execute.ts]"]
  node_tool_loop["LLM tool loop<br/>model executor<br/>[tool-loop.ts]"]
  node_tools["Agent tools<br/>tool registry<br/>[tools.ts]"]
  node_computer["Computer control<br/>privileged machine adapter<br/>[index.ts]"]
end

subgraph group_services["Services and policy"]
  node_models["Model providers<br/>provider layer<br/>[providers.ts]"]
  node_integrations["Workspace integrations<br/>external API adapter<br/>[google-oauth.ts]"]
  node_policy["Transaction policy<br/>approval gate"]
end

node_cli -->|"submits tasks"| node_agent
node_desktop -->|"local API"| node_gateway
node_gateway -->|"dispatches requests"| node_agent
node_telegram -->|"polling channel"| node_gateway
node_agent -->|"manages context"| node_sessions
node_sessions -->|"persists"| node_local_state
node_agent -->|"loads instructions"| node_skills
node_agent -->|"normalizes work"| node_orchestrator
node_orchestrator -->|"creates contract and plan"| node_planner
node_orchestrator -->|"enqueues execution"| node_step_runner
node_step_runner -->|"recognized workflows"| node_harness
node_step_runner -->|"open-ended workflows"| node_tool_loop
node_tool_loop -->|"model calls"| node_models
node_harness -->|"uses tools"| node_tools
node_tool_loop -->|"invokes tools"| node_tools
node_tools -->|"machine operations"| node_computer
node_tools -->|"service operations"| node_integrations
node_step_runner -->|"approval checks"| node_policy
node_step_runner -->|"records timeline"| node_local_state

click node_cli "https://github.com/safiyev/heyagent/blob/main/apps/cli/src/cli.ts"
click node_desktop "https://github.com/safiyev/heyagent/blob/main/apps/desktop/src/main.ts"
click node_gateway "https://github.com/safiyev/heyagent/blob/main/apps/gateway/src/index.ts"
click node_telegram "https://github.com/safiyev/heyagent/blob/main/packages/channels-telegram/src/index.ts"
click node_agent "https://github.com/safiyev/heyagent/blob/main/packages/agent/src/index.ts"
click node_sessions "https://github.com/safiyev/heyagent/blob/main/packages/agent/src/session.ts"
click node_skills "https://github.com/safiyev/heyagent/blob/main/packages/agent/src/skills-loader.ts"
click node_orchestrator "https://github.com/safiyev/heyagent/blob/main/packages/orchestrator/src/orchestrator.ts"
click node_planner "https://github.com/safiyev/heyagent/blob/main/packages/orchestrator/src/planner.ts"
click node_step_runner "https://github.com/safiyev/heyagent/blob/main/packages/orchestrator/src/step-runner.ts"
click node_harness "https://github.com/safiyev/heyagent/blob/main/packages/agent/src/harness/execute.ts"
click node_tool_loop "https://github.com/safiyev/heyagent/blob/main/packages/agent/src/tool-loop.ts"
click node_tools "https://github.com/safiyev/heyagent/blob/main/packages/agent/src/tools.ts"
click node_computer "https://github.com/safiyev/heyagent/blob/main/packages/computer/src/index.ts"
click node_models "https://github.com/safiyev/heyagent/blob/main/packages/models/src/providers.ts"
click node_integrations "https://github.com/safiyev/heyagent/blob/main/packages/integrations/src/google-oauth.ts"
click node_policy "https://github.com/safiyev/heyagent/blob/main/packages/policy/src/transaction-approvals.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_desktop,node_gateway,node_telegram toneBlue
class node_agent,node_sessions,node_local_state,node_skills toneAmber
class node_orchestrator,node_planner,node_step_runner,node_harness,node_tool_loop,node_tools,node_computer toneMint
class node_models,node_integrations,node_policy toneRose
```

## Components

### Local interfaces

- [CLI](https://github.com/safiyev/heyagent/blob/main/apps/cli/src/cli.ts): terminal adapter · Onboarding, configuration, tasks, and chat sessions. · `apps/cli/src/cli.ts`
- [Electron companion](https://github.com/safiyev/heyagent/blob/main/apps/desktop/src/main.ts): desktop UI · Companion UI communicates with the loopback gateway. · `apps/desktop/src/main.ts`
- [Local gateway](https://github.com/safiyev/heyagent/blob/main/apps/gateway/src/index.ts): localhost service · Long-running local API, jobs, missions, and Telegram boundary. · `apps/gateway/src/index.ts`
- [Telegram transport](https://github.com/safiyev/heyagent/blob/main/packages/channels-telegram/src/index.ts): channel adapter · `packages/channels-telegram/src/index.ts`

### Agent runtime

- [Agent runtime](https://github.com/safiyev/heyagent/blob/main/packages/agent/src/index.ts): runtime facade · Transforms requests into persistent agent work. · `packages/agent/src/index.ts`
- [Sessions and memory](https://github.com/safiyev/heyagent/blob/main/packages/agent/src/session.ts): state manager · Manages conversations, missions, queued sessions, and identity-backed memory. · `packages/agent/src/session.ts`
- **Local agent state**: persistent state · Sessions, memory, and mission history stored locally.
- [Skill loader](https://github.com/safiyev/heyagent/blob/main/packages/agent/src/skills-loader.ts): instruction loader · `packages/agent/src/skills-loader.ts`

### Execution

- [Orchestrator](https://github.com/safiyev/heyagent/blob/main/packages/orchestrator/src/orchestrator.ts): mission controller · Routes intent and runs resilient mission control flow. · `packages/orchestrator/src/orchestrator.ts`
- [Task planner](https://github.com/safiyev/heyagent/blob/main/packages/orchestrator/src/planner.ts): plan graph builder · Creates task contracts, plans, and verification criteria. · `packages/orchestrator/src/planner.ts`
- [Step runner](https://github.com/safiyev/heyagent/blob/main/packages/orchestrator/src/step-runner.ts): queued executor · Executes steps with retries, recovery, waits, and timeline persistence. · `packages/orchestrator/src/step-runner.ts`
- [Deterministic harness](https://github.com/safiyev/heyagent/blob/main/packages/agent/src/harness/execute.ts): workflow executor · Runs recognized workflows deterministically. · `packages/agent/src/harness/execute.ts`
- [LLM tool loop](https://github.com/safiyev/heyagent/blob/main/packages/agent/src/tool-loop.ts): model executor · Runs protected, compacted model-driven tool interactions. · `packages/agent/src/tool-loop.ts`
- [Agent tools](https://github.com/safiyev/heyagent/blob/main/packages/agent/src/tools.ts): tool registry · Desktop, integration, memory, scheduling, screen, and power tools. · `packages/agent/src/tools.ts`
- [Computer control](https://github.com/safiyev/heyagent/blob/main/packages/computer/src/index.ts): privileged machine adapter · Controls desktop, browser, files, media, and platform systems. · `packages/computer/src/index.ts`

### Services and policy

- [Model providers](https://github.com/safiyev/heyagent/blob/main/packages/models/src/providers.ts): provider layer · Selects, validates, credentials, and fails over model providers. · `packages/models/src/providers.ts`
- [Workspace integrations](https://github.com/safiyev/heyagent/blob/main/packages/integrations/src/google-oauth.ts): external API adapter · Connects Google OAuth, Gmail, Notion, and GitHub. · `packages/integrations/src/google-oauth.ts`
- [Transaction policy](https://github.com/safiyev/heyagent/blob/main/packages/policy/src/transaction-approvals.ts): approval gate · Gates capabilities and sensitive transaction approvals. · `packages/policy/src/transaction-approvals.ts`

## Connections

- CLI → Agent runtime: submits tasks
- Electron companion → Local gateway: local API
- Local gateway → Agent runtime: dispatches requests
- Telegram transport → Local gateway: polling channel
- Agent runtime → Sessions and memory: manages context
- Sessions and memory → Local agent state: persists
- Agent runtime → Skill loader: loads instructions
- Agent runtime → Orchestrator: normalizes work
- Orchestrator → Task planner: creates contract and plan
- Orchestrator → Step runner: enqueues execution
- Step runner → Deterministic harness: recognized workflows
- Step runner → LLM tool loop: open-ended workflows
- LLM tool loop → Model providers: model calls
- Deterministic harness → Agent tools: uses tools
- LLM tool loop → Agent tools: invokes tools
- Agent tools → Computer control: machine operations
- Agent tools → Workspace integrations: service operations
- Step runner → Transaction policy: approval checks
- Step runner → Local agent state: records timeline