safiyev/heyagent

Purpose and execution surfaces

HeyAgent is a local-first TypeScript/Node.js desktop automation agent. It accepts tasks through the terminal, Electron companion, or a locally hosted gateway with Telegram support. Primary entry points are apps/cli/src/cli.ts, apps/gateway/src/index.ts, and apps/desktop/src/main.ts; shared agent runtime starts at packages/agent/src/index.ts.

Interface adapters and local gateway

The CLI handles onboarding, configuration, one-shot tasks, and chat sessions. The Electron app is a companion UI that talks to the loopback gateway. The gateway is the long-running boundary for local API access, Telegram polling, scheduled jobs, and background missions; it should remain bound to localhost. Relevant paths: apps/cli/src/index.ts, apps/desktop/src/ui/app.js, apps/gateway/src/index.ts.

Agent context, sessions, and skills

The agent package owns conversation/session management, task normalization, identity-backed memory, mission history, queued sessions, chat commands, and dynamically loaded skill instructions. Its runtime converts user text into normalized work with persistent local state under ~/.heyagent. Relevant paths: packages/agent/src/session.ts, packages/agent/src/memory.ts, packages/agent/src/skills-loader.ts.

Deterministic orchestration pipeline

The orchestrator implements the core control flow: route an intent, create a task contract and plan graph, enqueue execution, run steps, record actions/world state, then verify criteria. Retry, recovery, waits, idempotency, and timeline persistence make mission execution resilient rather than a single model call. Relevant paths: packages/orchestrator/src/orchestrator.ts, packages/orchestrator/src/planner.ts, packages/orchestrator/src/step-runner.ts.

Harness/tool-loop execution boundary

Plans are dispatched either to deterministic harnesses for recognized workflows or to an LLM-driven tool loop. The agent supplies tool definitions for desktop, integrations, memory, scheduling, screen, and power operations; loop protections and compaction manage long-running model interactions. Relevant paths: packages/agent/src/harness/execute.ts, packages/agent/src/tool-loop.ts, packages/agent/src/tools.ts.

Desktop and browser control

packages/computer is the privileged local-machine adapter: screenshots/vision, mouse and keyboard input, windows/accessibility, browser navigation/profile access, files/shell-adjacent operations, media, and platform system controls. Windows has the richest implementation, with cross-platform alternatives where possible. Relevant paths: packages/computer/src/index.ts, packages/computer/src/browser-agent.ts, packages/computer/src/system-controls.ts.

Models, integrations, policy, and external services

The model layer selects providers, validates responses, manages credentials, and fails over among cloud, Bedrock/OpenRouter-compatible, or Ollama models. Integrations connect Google OAuth Workspace APIs, Gmail IMAP, Notion, and GitHub. Policy gates capabilities and transaction approvals before sensitive execution. Telegram transport and optional OpenAI Whisper STT/ElevenLabs TTS are external channels/services. Relevant paths: packages/models/src/providers.ts, packages/integrations/src/google-oauth.ts, packages/policy/src/transaction-approvals.ts, packages/channels-telegram/src/index.ts.