How simular-ai/agent-s works
Agent S is a framework for completing natural-language tasks through real desktop interfaces. The principal flow is task and screen observation into an agent, which plans and generates actions, grounds them against the GUI, and executes them through platform-specific control; the loop repeats until completion. Agent S3 adds code-agent grounding and Behavior Best-of-N judging, while earlier S1/S2 generations remain distinct implementations. The graph preserves core versioned modules and documented model/platform support. Internal wiring is based on indexed references and excerpts; unsampled details are omitted where relationships are uncertain.
Main entry points: worker.py S3 Worker · Manager.py S1 Manager · manager.py S2 Manager · grounding.py S2 Grounding
Components 34
Task Entry
Agent S1
- S1 Agent
gui_agents/s1/core/AgentS.py - S1 Manager
gui_agents/s1/core/Manager.py - S1 Worker
gui_agents/s1/core/Worker.py - S1 Knowledge
gui_agents/s1/core/Knowledge.py - S1 Procedures
gui_agents/s1/core/ProceduralMemory.py - S1 GUI Control
gui_agents/s1/aci/ACI.py - S1 Multimodal
gui_agents/s1/mllm/MultimodalAgent.py - S1 Model Engine
gui_agents/s1/mllm/MultimodalEngine.py
Agent S2
- S2 Manager
gui_agents/s2/agents/manager.py - S2 Grounding
gui_agents/s2/agents/grounding.py - S2 Worker
gui_agents/s2/agents/worker.py - S2 Knowledge
gui_agents/s2/core/knowledge.py - S2 Procedures
gui_agents/s2/memory/procedural_memory.py - S2 Multimodal
gui_agents/s2/core/mllm.py - S2 Model Engine
gui_agents/s2/core/engine.py
Agent S2.5
- S2.5 Agent
gui_agents/s2_5/agents/agent_s.py - S2.5 Grounding
gui_agents/s2_5/agents/grounding.py - S2.5 Worker
gui_agents/s2_5/agents/worker.py - S2.5 Multimodal
gui_agents/s2_5/core/mllm.py - S2.5 Model Engine
gui_agents/s2_5/core/engine.py
Agent S3
- S3 Agent
gui_agents/s3/agents/agent_s.py - S3 Grounding
gui_agents/s3/agents/grounding.py - S3 Worker
gui_agents/s3/agents/worker.py - Code Agent
gui_agents/s3/agents/code_agent.py - S3 Procedures
gui_agents/s3/memory/procedural_memory.py - S3 Multimodal
gui_agents/s3/core/mllm.py - S3 Model Engine
gui_agents/s3/core/engine.py - Behavior Narrator
gui_agents/s3/bbon/behavior_narrator.py - Comparative Judge
gui_agents/s3/bbon/comparative_judge.py
Other components
- Task User
Connections 37
- Task User → S3 CLI: submits task
- S3 CLI → S3 Agent: starts agent
- S3 Agent → S3 Worker: delegates work
- S3 Worker → S3 Grounding: requests grounding
gui_agents/s3/agents/worker.py - S3 Grounding → Code Agent: uses code agent
gui_agents/s3/agents/grounding.py - S3 Worker → S3 Procedures: uses procedures
gui_agents/s3/agents/worker.py - S3 Worker → S3 Multimodal: generates reasoning
- S3 Multimodal → S3 Model Engine: dispatches inference
gui_agents/s3/core/mllm.py - Task User → S1 CLI: submits task
- S1 CLI → S1 Agent: starts agent
- S1 Agent → S1 Manager: coordinates task
- S1 Manager → S1 Knowledge: retrieves context
gui_agents/s1/core/Manager.py - S1 Manager → S1 Worker: delegates actions
- S1 Manager → S1 GUI Control: controls interface
gui_agents/s1/core/Manager.py - S1 Manager → S1 Procedures: uses procedures
gui_agents/s1/core/Manager.py - S1 Knowledge → S1 Procedures: uses procedures
gui_agents/s1/core/Knowledge.py - S1 Worker → S1 GUI Control: uses controls
gui_agents/s1/core/Worker.py - Task User → S2 CLI: submits task
- S2 CLI → S2 Manager: starts manager
- S2 Manager → S2 Grounding: configures grounding
gui_agents/s2/agents/manager.py - S2 Manager → S2 Knowledge: retrieves context
gui_agents/s2/agents/manager.py - S2 Manager → S2 Procedures: uses procedures
gui_agents/s2/agents/manager.py - S2 Worker → S2 Grounding: requests grounding
gui_agents/s2/agents/worker.py - S2 Grounding → S2 Procedures: uses procedures
gui_agents/s2/agents/grounding.py - S2 Grounding → S2 Multimodal: requests inference
gui_agents/s2/agents/grounding.py - S2 Multimodal → S2 Model Engine: dispatches inference
gui_agents/s2/core/mllm.py - Task User → S2.5 CLI: submits task
- S2.5 CLI → S2.5 Agent: starts agent
- S2.5 Agent → S2.5 Worker: delegates work
gui_agents/s2_5/agents/agent_s.py - S2.5 Worker → S2.5 Grounding: requests grounding
gui_agents/s2_5/agents/worker.py - S2.5 Grounding → S2.5 Multimodal: requests inference
gui_agents/s2_5/agents/grounding.py - S2.5 Multimodal → S2.5 Model Engine: dispatches inference
gui_agents/s2_5/core/mllm.py - Task User → S3 CLI: submits task
- S3 Agent → S3 Grounding: selects grounding
- S3 Grounding → S3 Procedures: uses procedures
gui_agents/s3/agents/grounding.py - S3 Worker → Behavior Narrator: enables behavior selection
- S3 Worker → Comparative Judge: compares rollouts