# simular-ai/agent-s architecture

> Architecture diagram of the GitHub repository simular-ai/agent-s, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/simular-ai/agent-s
- Repository: https://github.com/simular-ai/agent-s

## Overview

Agent S is a framework for completing natural-language tasks through real desktop interfaces. The principal flow is task and screen observation into an agent, which plans and generates actions, grounds them against the GUI, and executes them through platform-specific control; the loop repeats until completion. Agent S3 adds code-agent grounding and Behavior Best-of-N judging, while earlier S1/S2 generations remain distinct implementations. The graph preserves core versioned modules and documented model/platform support. Internal wiring is based on indexed references and excerpts; unsampled details are omitted where relationships are uncertain.

Main entry points: [`gui_agents/s3/agents/worker.py`](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s3/agents/worker.py) (S3 Worker), [`gui_agents/s1/core/Manager.py`](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s1/core/Manager.py) (S1 Manager), [`gui_agents/s2/agents/manager.py`](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2/agents/manager.py) (S2 Manager), [`gui_agents/s2/agents/grounding.py`](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2/agents/grounding.py) (S2 Grounding)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Task Entry"]
  node_cli1["S1 CLI<br/>[cli_app.py]"]
  node_cli2["S2 CLI<br/>[cli_app.py]"]
  node_cli25["S2.5 CLI<br/>[cli_app.py]"]
  node_cli3["S3 CLI<br/>[cli_app.py]"]
end

subgraph group_s1["Agent S1"]
  node_agent1["S1 Agent<br/>[AgentS.py]"]
  node_manager1["S1 Manager<br/>[Manager.py]"]
  node_worker1["S1 Worker<br/>[Worker.py]"]
  node_knowledge1[("S1 Knowledge<br/>[Knowledge.py]")]
  node_memory1["S1 Procedures"]
  node_aci1["S1 GUI Control<br/>[ACI.py]"]
  node_multimodal1["S1 Multimodal<br/>[MultimodalAgent.py]"]
  node_engine1["S1 Model Engine"]
end

subgraph group_s2["Agent S2"]
  node_manager2["S2 Manager<br/>[manager.py]"]
  node_ground2["S2 Grounding<br/>[grounding.py]"]
  node_worker2["S2 Worker<br/>[worker.py]"]
  node_knowledge2[("S2 Knowledge<br/>[knowledge.py]")]
  node_memory2["S2 Procedures"]
  node_mllm2["S2 Multimodal<br/>[mllm.py]"]
  node_engine2["S2 Model Engine<br/>[engine.py]"]
end

subgraph group_s25["Agent S2.5"]
  node_agent25["S2.5 Agent<br/>[agent_s.py]"]
  node_ground25["S2.5 Grounding<br/>[grounding.py]"]
  node_worker25["S2.5 Worker<br/>[worker.py]"]
  node_mllm25["S2.5 Multimodal<br/>[mllm.py]"]
  node_engine25["S2.5 Model Engine<br/>[engine.py]"]
end

subgraph group_s3["Agent S3"]
  node_agent3["S3 Agent<br/>[agent_s.py]"]
  node_ground3["S3 Grounding<br/>[grounding.py]"]
  node_worker3["S3 Worker<br/>[worker.py]"]
  node_code3["Code Agent<br/>[code_agent.py]"]
  node_memory3["S3 Procedures"]
  node_mllm3["S3 Multimodal<br/>[mllm.py]"]
  node_engine3["S3 Model Engine<br/>[engine.py]"]
  node_narrator3["Behavior Narrator"]
  node_judge3["Comparative Judge"]
end

node_user(("Task User"))

node_user -.->|"submits task"| node_cli3
node_cli3 -.->|"starts agent"| node_agent3
node_agent3 -->|"delegates work"| node_worker3
node_worker3 -->|"requests grounding"| node_ground3
node_ground3 -->|"uses code agent"| node_code3
node_worker3 -->|"uses procedures"| node_memory3
node_worker3 -.->|"generates reasoning"| node_mllm3
node_mllm3 -->|"dispatches inference"| node_engine3
node_user -.->|"submits task"| node_cli1
node_cli1 -.->|"starts agent"| node_agent1
node_agent1 -.->|"coordinates task"| node_manager1
node_manager1 -->|"retrieves context"| node_knowledge1
node_manager1 -.->|"delegates actions"| node_worker1
node_manager1 -.->|"controls interface"| node_aci1
node_manager1 -->|"uses procedures"| node_memory1
node_knowledge1 -->|"uses procedures"| node_memory1
node_worker1 -->|"uses controls"| node_aci1
node_user -.->|"submits task"| node_cli2
node_cli2 -.->|"starts manager"| node_manager2
node_manager2 -->|"configures grounding"| node_ground2
node_manager2 -->|"retrieves context"| node_knowledge2
node_manager2 -->|"uses procedures"| node_memory2
node_worker2 -->|"requests grounding"| node_ground2
node_ground2 -->|"uses procedures"| node_memory2
node_ground2 -->|"requests inference"| node_mllm2
node_mllm2 -->|"dispatches inference"| node_engine2
node_user -.->|"submits task"| node_cli25
node_cli25 -.->|"starts agent"| node_agent25
node_agent25 -->|"delegates work"| node_worker25
node_worker25 -->|"requests grounding"| node_ground25
node_ground25 -->|"requests inference"| node_mllm25
node_mllm25 -->|"dispatches inference"| node_engine25
node_user -.->|"submits task"| node_cli3
node_agent3 -->|"selects grounding"| node_ground3
node_ground3 -->|"uses procedures"| node_memory3
node_worker3 -.->|"enables behavior selection"| node_narrator3
node_worker3 -.->|"compares rollouts"| node_judge3

click node_cli1 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s1/cli_app.py"
click node_cli2 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2/cli_app.py"
click node_cli25 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2_5/cli_app.py"
click node_cli3 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s3/cli_app.py"
click node_agent1 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s1/core/AgentS.py"
click node_manager1 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s1/core/Manager.py"
click node_worker1 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s1/core/Worker.py"
click node_knowledge1 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s1/core/Knowledge.py"
click node_memory1 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s1/core/ProceduralMemory.py"
click node_aci1 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s1/aci/ACI.py"
click node_multimodal1 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s1/mllm/MultimodalAgent.py"
click node_engine1 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s1/mllm/MultimodalEngine.py"
click node_manager2 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2/agents/manager.py"
click node_ground2 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2/agents/grounding.py"
click node_worker2 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2/agents/worker.py"
click node_knowledge2 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2/core/knowledge.py"
click node_memory2 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2/memory/procedural_memory.py"
click node_mllm2 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2/core/mllm.py"
click node_engine2 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2/core/engine.py"
click node_agent25 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2_5/agents/agent_s.py"
click node_ground25 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2_5/agents/grounding.py"
click node_worker25 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2_5/agents/worker.py"
click node_mllm25 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2_5/core/mllm.py"
click node_engine25 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2_5/core/engine.py"
click node_agent3 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s3/agents/agent_s.py"
click node_ground3 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s3/agents/grounding.py"
click node_worker3 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s3/agents/worker.py"
click node_code3 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s3/agents/code_agent.py"
click node_memory3 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s3/memory/procedural_memory.py"
click node_mllm3 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s3/core/mllm.py"
click node_engine3 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s3/core/engine.py"
click node_narrator3 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s3/bbon/behavior_narrator.py"
click node_judge3 "https://github.com/simular-ai/agent-s/blob/main/gui_agents/s3/bbon/comparative_judge.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli1,node_cli2,node_cli25,node_cli3 toneBlue
class node_agent1,node_manager1,node_worker1,node_knowledge1,node_memory1,node_aci1,node_multimodal1,node_engine1 toneAmber
class node_manager2,node_ground2,node_worker2,node_knowledge2,node_memory2,node_mllm2,node_engine2 toneMint
class node_agent25,node_ground25,node_worker25,node_mllm25,node_engine25,node_user toneRose
class node_agent3,node_ground3,node_worker3,node_code3,node_memory3,node_mllm3,node_engine3,node_narrator3,node_judge3 toneIndigo
```

## Components

### Task Entry

- [S1 CLI](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s1/cli_app.py): `gui_agents/s1/cli_app.py`
- [S2 CLI](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2/cli_app.py): `gui_agents/s2/cli_app.py`
- [S2.5 CLI](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2_5/cli_app.py): `gui_agents/s2_5/cli_app.py`
- [S3 CLI](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s3/cli_app.py): `gui_agents/s3/cli_app.py`

### Agent S1

- [S1 Agent](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s1/core/AgentS.py): `gui_agents/s1/core/AgentS.py`
- [S1 Manager](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s1/core/Manager.py): `gui_agents/s1/core/Manager.py`
- [S1 Worker](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s1/core/Worker.py): `gui_agents/s1/core/Worker.py`
- [S1 Knowledge](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s1/core/Knowledge.py): `gui_agents/s1/core/Knowledge.py`
- [S1 Procedures](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s1/core/ProceduralMemory.py): `gui_agents/s1/core/ProceduralMemory.py`
- [S1 GUI Control](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s1/aci/ACI.py): `gui_agents/s1/aci/ACI.py`
- [S1 Multimodal](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s1/mllm/MultimodalAgent.py): `gui_agents/s1/mllm/MultimodalAgent.py`
- [S1 Model Engine](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s1/mllm/MultimodalEngine.py): `gui_agents/s1/mllm/MultimodalEngine.py`

### Agent S2

- [S2 Manager](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2/agents/manager.py): `gui_agents/s2/agents/manager.py`
- [S2 Grounding](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2/agents/grounding.py): `gui_agents/s2/agents/grounding.py`
- [S2 Worker](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2/agents/worker.py): `gui_agents/s2/agents/worker.py`
- [S2 Knowledge](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2/core/knowledge.py): `gui_agents/s2/core/knowledge.py`
- [S2 Procedures](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2/memory/procedural_memory.py): `gui_agents/s2/memory/procedural_memory.py`
- [S2 Multimodal](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2/core/mllm.py): `gui_agents/s2/core/mllm.py`
- [S2 Model Engine](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2/core/engine.py): `gui_agents/s2/core/engine.py`

### Agent S2.5

- [S2.5 Agent](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2_5/agents/agent_s.py): `gui_agents/s2_5/agents/agent_s.py`
- [S2.5 Grounding](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2_5/agents/grounding.py): `gui_agents/s2_5/agents/grounding.py`
- [S2.5 Worker](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2_5/agents/worker.py): `gui_agents/s2_5/agents/worker.py`
- [S2.5 Multimodal](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2_5/core/mllm.py): `gui_agents/s2_5/core/mllm.py`
- [S2.5 Model Engine](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s2_5/core/engine.py): `gui_agents/s2_5/core/engine.py`

### Agent S3

- [S3 Agent](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s3/agents/agent_s.py): `gui_agents/s3/agents/agent_s.py`
- [S3 Grounding](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s3/agents/grounding.py): `gui_agents/s3/agents/grounding.py`
- [S3 Worker](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s3/agents/worker.py): `gui_agents/s3/agents/worker.py`
- [Code Agent](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s3/agents/code_agent.py): `gui_agents/s3/agents/code_agent.py`
- [S3 Procedures](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s3/memory/procedural_memory.py): `gui_agents/s3/memory/procedural_memory.py`
- [S3 Multimodal](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s3/core/mllm.py): `gui_agents/s3/core/mllm.py`
- [S3 Model Engine](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s3/core/engine.py): `gui_agents/s3/core/engine.py`
- [Behavior Narrator](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s3/bbon/behavior_narrator.py): `gui_agents/s3/bbon/behavior_narrator.py`
- [Comparative Judge](https://github.com/simular-ai/agent-s/blob/main/gui_agents/s3/bbon/comparative_judge.py): `gui_agents/s3/bbon/comparative_judge.py`

### Other components

- **Task User**

## Connections

- Task User → S3 CLI: submits task
- S3 CLI → S3 Agent: starts agent
- S3 Agent → S3 Worker: delegates work
- S3 Worker → S3 Grounding: requests grounding (evidence: `gui_agents/s3/agents/worker.py`)
- S3 Grounding → Code Agent: uses code agent (evidence: `gui_agents/s3/agents/grounding.py`)
- S3 Worker → S3 Procedures: uses procedures (evidence: `gui_agents/s3/agents/worker.py`)
- S3 Worker → S3 Multimodal: generates reasoning
- S3 Multimodal → S3 Model Engine: dispatches inference (evidence: `gui_agents/s3/core/mllm.py`)
- Task User → S1 CLI: submits task
- S1 CLI → S1 Agent: starts agent
- S1 Agent → S1 Manager: coordinates task
- S1 Manager → S1 Knowledge: retrieves context (evidence: `gui_agents/s1/core/Manager.py`)
- S1 Manager → S1 Worker: delegates actions
- S1 Manager → S1 GUI Control: controls interface (evidence: `gui_agents/s1/core/Manager.py`)
- S1 Manager → S1 Procedures: uses procedures (evidence: `gui_agents/s1/core/Manager.py`)
- S1 Knowledge → S1 Procedures: uses procedures (evidence: `gui_agents/s1/core/Knowledge.py`)
- S1 Worker → S1 GUI Control: uses controls (evidence: `gui_agents/s1/core/Worker.py`)
- Task User → S2 CLI: submits task
- S2 CLI → S2 Manager: starts manager
- S2 Manager → S2 Grounding: configures grounding (evidence: `gui_agents/s2/agents/manager.py`)
- S2 Manager → S2 Knowledge: retrieves context (evidence: `gui_agents/s2/agents/manager.py`)
- S2 Manager → S2 Procedures: uses procedures (evidence: `gui_agents/s2/agents/manager.py`)
- S2 Worker → S2 Grounding: requests grounding (evidence: `gui_agents/s2/agents/worker.py`)
- S2 Grounding → S2 Procedures: uses procedures (evidence: `gui_agents/s2/agents/grounding.py`)
- S2 Grounding → S2 Multimodal: requests inference (evidence: `gui_agents/s2/agents/grounding.py`)
- S2 Multimodal → S2 Model Engine: dispatches inference (evidence: `gui_agents/s2/core/mllm.py`)
- Task User → S2.5 CLI: submits task
- S2.5 CLI → S2.5 Agent: starts agent
- S2.5 Agent → S2.5 Worker: delegates work (evidence: `gui_agents/s2_5/agents/agent_s.py`)
- S2.5 Worker → S2.5 Grounding: requests grounding (evidence: `gui_agents/s2_5/agents/worker.py`)
- S2.5 Grounding → S2.5 Multimodal: requests inference (evidence: `gui_agents/s2_5/agents/grounding.py`)
- S2.5 Multimodal → S2.5 Model Engine: dispatches inference (evidence: `gui_agents/s2_5/core/mllm.py`)
- Task User → S3 CLI: submits task
- S3 Agent → S3 Grounding: selects grounding
- S3 Grounding → S3 Procedures: uses procedures (evidence: `gui_agents/s3/agents/grounding.py`)
- S3 Worker → Behavior Narrator: enables behavior selection
- S3 Worker → Comparative Judge: compares rollouts
