# aidriventesting/robotframework-agent architecture

> Architecture diagram of the GitHub repository aidriventesting/robotframework-agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aidriventesting/robotframework-agent
- Repository: https://github.com/aidriventesting/robotframework-agent

## Overview

Robot Framework Agent turns natural-language UI instructions into LLM-selected tools, executes them against a UI platform, and supports screen questions, checks, and visual element discovery. The main runtime path runs from Robot Framework keywords through the agent engine, prompt/tool preparation, LLM response, and tool execution. Mobile context collection and visual parsing provide distinct perception paths. Several implementation details are unsampled, so relationships not established by the README or cited source index are omitted.

Main entry points: [`Agent/agent_engine.py`](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/agent_engine.py) (Agent engine), [`Agent/ai/llm/_factory.py`](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/ai/llm/_factory.py) (Provider factory), [`Agent/platforms/_mobileconnector.py`](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/platforms/_mobileconnector.py) (Mobile connector), [`Agent/ai/vlm/interface.py`](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/ai/vlm/interface.py) (Visual parsing interface)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Robot interface"]
  node_keywords["Agent keywords<br/>[AgentKeywords.py]"]
end

subgraph group_reasoning["Agent reasoning"]
  node_engine["Agent engine<br/>[agent_engine.py]"]
  node_composer["Prompt composer<br/>[_promptcomposer.py]"]
  node_llm["LLM facade<br/>[facade.py]"]
  node_llmfactory["Provider factory<br/>[_factory.py]"]
  node_openai["OpenAI client<br/>[_openaiclient.py]"]
  node_anthropic["Anthropic client<br/>[_anthropic.py]"]
  node_gemini["Gemini client<br/>[_gemini.py]"]
end

subgraph group_perception["UI perception"]
  node_connector["Mobile connector"]
  node_collectors["UI collectors"]
  node_filters["Element filters<br/>[pipeline.py]"]
  node_grounding["UI grounding"]
  node_vlminterface["Visual parsing interface<br/>[interface.py]"]
  node_vlmclient["OmniParser client<br/>[_client.py]"]
  node_vlmparser["OmniParser result parser<br/>[_parser.py]"]
end

subgraph group_execution["Tools and platforms"]
  node_registry["Tool registry<br/>[registry.py]"]
  node_toolbase["Tool interface<br/>[base.py]"]
  node_mobiletools["Mobile actions"]
  node_screentools["Screen answers and checks"]
end

subgraph group_support["Configuration and usage"]
  node_config["Runtime configuration<br/>[config.py]"]
  node_modelconfig["Model configuration<br/>[model_config.py]"]
  node_cost["LLM cost tracking<br/>[_costtracker.py]"]
  node_pricing["Model price lookup<br/>[_pricefetcher.py]"]
  node_imageupload["Image uploader<br/>[imghandler.py]"]
end

node_robot(("Robot Framework"))
node_omniparser["OmniParser service"]

node_robot -->|"invokes keywords"| node_keywords
node_keywords -->|"delegates actions"| node_engine
node_engine -->|"builds prompts"| node_composer
node_engine -->|"requests completions"| node_llm
node_engine -->|"selects tools"| node_registry
node_engine -->|"registers screen tools"| node_screentools
node_engine -->|"registers mobile tools"| node_mobiletools
node_llm -->|"creates clients"| node_llmfactory
node_llmfactory -.->|"selects provider"| node_openai
node_llmfactory -.->|"selects provider"| node_anthropic
node_llmfactory -.->|"selects provider"| node_gemini
node_registry -->|"stores tools"| node_toolbase
node_mobiletools -->|"implements interface"| node_toolbase
node_screentools -->|"implements interface"| node_toolbase
node_engine -->|"creates platform"| node_connector
node_connector -->|"parses UI"| node_collectors
node_connector -->|"filters candidates"| node_filters
node_connector -->|"serializes UI context"| node_grounding
node_engine -->|"requests visual parsing"| node_vlminterface
node_vlminterface -->|"calls parser service"| node_vlmclient
node_vlminterface -->|"parses response"| node_vlmparser
node_vlmclient -->|"sends image request"| node_omniparser
node_config -->|"loads model defaults"| node_modelconfig
node_modelconfig -->|"initializes price lookup"| node_pricing
node_openai -->|"records usage cost"| node_cost
node_engine -->|"uploads images"| node_imageupload
node_imageupload -->|"reads upload settings"| node_config

click node_keywords "https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/AgentKeywords.py"
click node_engine "https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/agent_engine.py"
click node_composer "https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/ai/_promptcomposer.py"
click node_llm "https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/ai/llm/facade.py"
click node_llmfactory "https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/ai/llm/_factory.py"
click node_openai "https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/ai/llm/_openaiclient.py"
click node_anthropic "https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/ai/llm/_anthropic.py"
click node_gemini "https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/ai/llm/_gemini.py"
click node_registry "https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/tools/registry.py"
click node_toolbase "https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/tools/base.py"
click node_mobiletools "https://github.com/aidriventesting/robotframework-agent/tree/main/Agent/tools/mobile"
click node_screentools "https://github.com/aidriventesting/robotframework-agent/tree/main/Agent/tools/screen"
click node_connector "https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/platforms/_mobileconnector.py"
click node_collectors "https://github.com/aidriventesting/robotframework-agent/tree/main/Agent/platforms/collectors"
click node_filters "https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/platforms/filters/pipeline.py"
click node_grounding "https://github.com/aidriventesting/robotframework-agent/tree/main/Agent/platforms/grounding"
click node_vlminterface "https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/ai/vlm/interface.py"
click node_vlmclient "https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/ai/vlm/_client.py"
click node_vlmparser "https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/ai/vlm/_parser.py"
click node_config "https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/config/config.py"
click node_modelconfig "https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/config/model_config.py"
click node_cost "https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/utilities/_costtracker.py"
click node_pricing "https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/utilities/_pricefetcher.py"
click node_imageupload "https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/utilities/imguploader/imghandler.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_keywords toneBlue
class node_engine,node_composer,node_llm,node_llmfactory,node_openai,node_anthropic,node_gemini toneAmber
class node_connector,node_collectors,node_filters,node_grounding,node_vlminterface,node_vlmclient,node_vlmparser toneMint
class node_registry,node_toolbase,node_mobiletools,node_screentools toneRose
class node_config,node_modelconfig,node_cost,node_pricing,node_imageupload,node_robot,node_omniparser toneIndigo
```

## Components

### Robot interface

- [Agent keywords](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/AgentKeywords.py): `Agent/AgentKeywords.py`

### Agent reasoning

- [Agent engine](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/agent_engine.py): `Agent/agent_engine.py`
- [Prompt composer](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/ai/_promptcomposer.py): `Agent/ai/_promptcomposer.py`
- [LLM facade](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/ai/llm/facade.py): `Agent/ai/llm/facade.py`
- [Provider factory](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/ai/llm/_factory.py): `Agent/ai/llm/_factory.py`
- [OpenAI client](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/ai/llm/_openaiclient.py): `Agent/ai/llm/_openaiclient.py`
- [Anthropic client](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/ai/llm/_anthropic.py): `Agent/ai/llm/_anthropic.py`
- [Gemini client](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/ai/llm/_gemini.py): `Agent/ai/llm/_gemini.py`

### UI perception

- [Mobile connector](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/platforms/_mobileconnector.py): `Agent/platforms/_mobileconnector.py`
- [UI collectors](https://github.com/aidriventesting/robotframework-agent/tree/main/Agent/platforms/collectors): `Agent/platforms/collectors`
- [Element filters](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/platforms/filters/pipeline.py): `Agent/platforms/filters/pipeline.py`
- [UI grounding](https://github.com/aidriventesting/robotframework-agent/tree/main/Agent/platforms/grounding): `Agent/platforms/grounding`
- [Visual parsing interface](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/ai/vlm/interface.py): `Agent/ai/vlm/interface.py`
- [OmniParser client](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/ai/vlm/_client.py): `Agent/ai/vlm/_client.py`
- [OmniParser result parser](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/ai/vlm/_parser.py): `Agent/ai/vlm/_parser.py`

### Tools and platforms

- [Tool registry](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/tools/registry.py): `Agent/tools/registry.py`
- [Tool interface](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/tools/base.py): `Agent/tools/base.py`
- [Mobile actions](https://github.com/aidriventesting/robotframework-agent/tree/main/Agent/tools/mobile): `Agent/tools/mobile`
- [Screen answers and checks](https://github.com/aidriventesting/robotframework-agent/tree/main/Agent/tools/screen): `Agent/tools/screen`

### Configuration and usage

- [Runtime configuration](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/config/config.py): `Agent/config/config.py`
- [Model configuration](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/config/model_config.py): `Agent/config/model_config.py`
- [LLM cost tracking](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/utilities/_costtracker.py): `Agent/utilities/_costtracker.py`
- [Model price lookup](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/utilities/_pricefetcher.py): `Agent/utilities/_pricefetcher.py`
- [Image uploader](https://github.com/aidriventesting/robotframework-agent/blob/main/Agent/utilities/imguploader/imghandler.py): `Agent/utilities/imguploader/imghandler.py`

### Other components

- **Robot Framework**
- **OmniParser service**

## Connections

- Robot Framework → Agent keywords: invokes keywords
- Agent keywords → Agent engine: delegates actions (evidence: `Agent/AgentKeywords.py`)
- Agent engine → Prompt composer: builds prompts (evidence: `Agent/agent_engine.py`)
- Agent engine → LLM facade: requests completions (evidence: `Agent/agent_engine.py`)
- Agent engine → Tool registry: selects tools (evidence: `Agent/agent_engine.py`)
- Agent engine → Screen answers and checks: registers screen tools (evidence: `Agent/agent_engine.py`)
- Agent engine → Mobile actions: registers mobile tools (evidence: `Agent/agent_engine.py`)
- LLM facade → Provider factory: creates clients (evidence: `Agent/ai/llm/facade.py`)
- Provider factory → OpenAI client: selects provider
- Provider factory → Anthropic client: selects provider
- Provider factory → Gemini client: selects provider
- Tool registry → Tool interface: stores tools (evidence: `Agent/tools/registry.py`)
- Mobile actions → Tool interface: implements interface (evidence: `Agent/tools/mobile/click_element.py`)
- Screen answers and checks → Tool interface: implements interface (evidence: `Agent/tools/screen/answer_text.py`)
- Agent engine → Mobile connector: creates platform (evidence: `Agent/agent_engine.py`)
- Mobile connector → UI collectors: parses UI (evidence: `Agent/platforms/_mobileconnector.py`)
- Mobile connector → Element filters: filters candidates (evidence: `Agent/platforms/_mobileconnector.py`)
- Mobile connector → UI grounding: serializes UI context (evidence: `Agent/platforms/_mobileconnector.py`)
- Agent engine → Visual parsing interface: requests visual parsing (evidence: `Agent/agent_engine.py`)
- Visual parsing interface → OmniParser client: calls parser service (evidence: `Agent/ai/vlm/interface.py`)
- Visual parsing interface → OmniParser result parser: parses response (evidence: `Agent/ai/vlm/interface.py`)
- OmniParser client → OmniParser service: sends image request
- Runtime configuration → Model configuration: loads model defaults (evidence: `Agent/config/config.py`)
- Model configuration → Model price lookup: initializes price lookup (evidence: `Agent/config/model_config.py`)
- OpenAI client → LLM cost tracking: records usage cost (evidence: `Agent/ai/llm/_openaiclient.py`)
- Agent engine → Image uploader: uploads images (evidence: `Agent/agent_engine.py`)
- Image uploader → Runtime configuration: reads upload settings (evidence: `Agent/utilities/imguploader/imghandler.py`)