How andokivilaid/omnigent works
1. This is a multi-surface AI orchestration platform, not a single app.
Omnigent combines a Python backend/orchestrator, a React/TypeScript web UI, a terminal/CLI experience, desktop wrappers, and a set of agent harnesses that connect to external LLM providers and local tools.
2. User-facing surfaces are split into three main entry points.
- omnigent / omni CLI for terminal-first sessions and host registration
- ap-web for the browser UI
- Electron packaging under ap-web/electron for the macOS desktop app that wraps the same web UI and adds OS integrations
3. The core control plane lives in omnigent/server and omnigent/runner.
The server owns auth, sessions, hosts, policies, comments, permissions, and persistence-facing APIs.
The runner is the execution side: it hosts agent sessions, dispatches tools, manages approvals, and brokers runtime communication back to the server.
4. Session state is central and persistent.
Conversations, agents, files, comments, permissions, policies, hosts, and related metadata are stored through SQLAlchemy-backed stores in omnigent/stores with schema and migrations in omnigent/db.
The server exposes these resources through HTTP APIs and WebSocket/SSE streams so the web UI and CLI can observe live updates.
5. Agent execution is abstracted behind harnesses and executors.
omnigent/inner contains the concrete harness integrations for Claude, Codex, Cursor, Pi, OpenAI Agents SDK, Antigravity, Databricks, and other modes.
This layer is where the platform normalizes different agent runtimes into a common execution model.
6. Policies are first-class and enforced in the runtime path.
Policy definitions live under omnigent/policies and runtime enforcement under omnigent/runtime/policies.
They gate shell access, file writes, tool calls, routing, and spending decisions before actions are executed, so policy checks sit on the critical path between agent intent and tool execution.
7. Tools are a separate subsystem from agents.
omnigent/tools defines built-in tools, client-specified tools, MCP integration, and tool dispatch/validation.
Tools are invoked by the runner/runtime, can be local Python callables or external tool servers, and are surfaced back into conversations as structured events.
8. Sandboxing and host execution are explicit boundaries.
Local and cloud execution environments are handled in omnigent/sandbox, omnigent/host, and omnigent/onboarding/sandboxes.
The repo supports macOS seatbelt, Linux bubblewrap, and managed/cloud sandboxes such as Modal, Daytona, Islo, Cloud Workstations, Kubernetes, and Docker-based deployments.
This boundary isolates agent work from the user machine and from the server process.
9. Model/provider selection is decoupled from orchestration.
omnigent/llms contains provider adapters for Anthropic, OpenAI, Gemini, Bedrock, Vertex, Databricks, and gateway-style endpoints.
The onboarding flow and model catalog under omnigent/onboarding and omnigent/model_catalog.py choose credentials and defaults, while the runtime consumes a provider-neutral LLM client.
10. The web app is a live session console, not just a chat UI.
ap-web/src is organized around pages, shell panels, chat blocks, file viewers, terminal views, comments, approvals, and agent controls.
It consumes the server’s session streams to render conversation updates, sub-agent activity, terminal output, file diffs, and policy prompts in real time.
11. Live collaboration is built into the architecture.
The system supports shared sessions, co-driving, forks, and multi-user presence.
That shows up in server routes for presence/auth/session sharing, runner transports, and UI features for session switching, comments, and approvals.
12. Transport is layered to support local, remote, and tunneled execution.
The runner exposes transports over TCP, UDS, and WebSocket tunnels in omnigent/runner/transports, while the server has tunnel routes for hosts and runners.
This allows the same session model to work whether the agent is local on a laptop or running on a managed remote host.
13. Deployment is multi-target and configuration-driven.
The deploy/ tree contains Docker, Kubernetes, Fly, Railway, Render, Modal, Hugging Face Spaces, and Daytona/CWSandbox setups.
The repo is designed so the same server/runner architecture can be deployed in several environments with different auth, database, and sandbox backends.
14. SDKs and examples are secondary integration surfaces.
sdks/python-client exposes the server API to external programs, while sdks/ui provides terminal-rendering helpers.
examples/ and tests/resources/ demonstrate how to define YAML agents, skills, sub-agents, tools, and policy stacks that plug into the same orchestration pipeline.
15. A useful diagram should show four main zones and the arrows between them.
- Client surfaces: CLI, web UI, Electron
- Control plane: server, auth, persistence, collaboration APIs
- Execution plane: runner, runtime, policies, tools, transports
- External dependencies: model providers, sandboxes, databases, OIDC/auth, MCP/tool servers
The key flows are user input → session/event stream → policy/runtime enforcement → harness/tool execution → persisted session updates → UI/CLI refresh.
Main entry points: server/app.py Server · runner/app.py Runner · runtime Runtime · src Web UI
Components 19
Client surfaces
Terminal, web, and desktop entry points for users.
- CLI · Terminal-first command surface for sessions, host registration, and REPL flows.
omnigent/cli.py - Web UI · Browser-based live session console with chat, files, approvals, comments, and terminals.
ap-web/src - Desktop app · Electron packaging around the web UI with macOS integration.
ap-web/electron/src
Control plane
Server APIs, auth, collaboration, and session persistence boundaries.
- Server · Central control plane for auth, sessions, hosts, permissions, comments, and collaboration streams.
omnigent/server/app.py - Auth · Login, account bootstrap, OIDC, and access control plumbing.
omnigent/server/auth.py - Sessions API · Session CRUD, sharing, forks, state updates, and live event endpoints.
omnigent/server/routes/sessions.py - Collaboration · Presence, comments, approvals, and multi-user collaboration routes.
omnigent/server/routes/comments.py - Host API · Host registry, tunnel endpoints, worktree launch, and remote runner connections.
omnigent/server/routes/hosts.py
Execution plane
Runner, runtime, policies, tools, harnesses, and transports.
- Runner · Runtime execution side that hosts agent sessions, dispatches tools, and brokers communication.
omnigent/runner/app.py - Runtime · Workflow, compaction, prompts, streaming, approvals, and runtime orchestration logic.
omnigent/runtime - Policies · Policy definitions and runtime enforcement for shell, file, tool, routing, and spending decisions.
omnigent/runtime/policies - Tools · Built-in, client-specified, local, and MCP-backed tools with validation and dispatch.
omnigent/tools - Harnesses · Adapters that normalize Claude, Codex, Cursor, Pi, OpenAI Agents SDK, Antigravity, and Databricks execution modes.
omnigent/inner - Transports · TCP, UDS, and WebSocket tunnel transports between server, runner, and hosts.
omnigent/runner/transports
Persistence
Database-backed stores and schema used for session state.
External systems
Model providers, sandboxes, auth, and tool servers outside the repo.
- LLM adapters · Provider-neutral adapters for Anthropic, OpenAI, Gemini, Bedrock, Vertex, Databricks, and gateways.
omnigent/llms/adapters - Sandboxes · Local and managed sandbox backends such as seatbelt, bubblewrap, Modal, Daytona, Islo, CWSandbox, Kubernetes, and Docker.
omnigent/sandbox - External tools · Remote MCP servers, local tool callables, and other external tool execution targets.
omnigent/tools/mcp.py
Connections 21
- CLI → Server: session API · CLI reads and mutates sessions through the server and subscribes to updates.
- Web UI → Server: live console · Web UI consumes HTTP APIs plus streaming updates from the server.
- Desktop app → Web UI: wraps UI · Electron hosts the same web application and adds native integration.
- Server → Auth: authenticate · Server delegates login and access control to auth and OIDC flows.
- Server → Sessions API: session routes · Session endpoints expose conversation state, forks, and live streams.
- Server → Collaboration: collaboration · Server exposes presence, comments, approvals, and sharing features.
- Server → Host API: host registry · Server manages hosts, tunnels, and remote runner attachment.
- Server → Stores: persist state · Server reads and writes canonical session resources through stores.
- Stores → DB schema: schema/migrations · Stores are backed by SQLAlchemy models and Alembic migrations.
- Runner → Runtime: execute workflow · Runner drives session workflows and emits structured session events.
- Runtime → Policies: enforce · Runtime gates actions through policy checks before execution.
- Runtime → Tools: invoke tools · Runtime dispatches tool calls and forwards structured results.
- Runtime → Harnesses: spawn agents · Runtime uses harness adapters to run different agent runtimes.
- Runner → Transports: connect · Runner exposes transports for local, remote, and tunneled communication.
- Runner → Server: stream events · Runner reports session progress and accepts control messages from server routes.
- Harnesses → LLM adapters: model calls · Harnesses use provider-neutral LLM clients via adapters.
- Harnesses → Sandboxes: isolate work · Harness execution is bound to sandboxed local or managed environments.
- Tools → External tools: MCP/local · Tools can dispatch to local callables or external MCP servers.
- Server → Runner: tunnel/attach · Server coordinates runner attachment and tunneled session connections.
- Web UI → Sessions API: stream updates · UI subscribes to live session and collaboration events for rendering.
- CLI → Runner: terminal session · CLI can attach to or launch execution-side sessions through the runner.