How aidriventesting/robotframework-agent works
Robot Framework Agent turns natural-language UI instructions into LLM-selected tools, executes them against a UI platform, and supports screen questions, checks, and visual element discovery. The main runtime path runs from Robot Framework keywords through the agent engine, prompt/tool preparation, LLM response, and tool execution. Mobile context collection and visual parsing provide distinct perception paths. Several implementation details are unsampled, so relationships not established by the README or cited source index are omitted.
Main entry points: agent_engine.py Agent engine · _factory.py Provider factory · _mobileconnector.py Mobile connector · interface.py Visual parsing interface
Components 26
Robot interface
- Agent keywords
Agent/AgentKeywords.py
Agent reasoning
- Agent engine
Agent/agent_engine.py - Prompt composer
Agent/ai/_promptcomposer.py - LLM facade
Agent/ai/llm/facade.py - Provider factory
Agent/ai/llm/_factory.py - OpenAI client
Agent/ai/llm/_openaiclient.py - Anthropic client
Agent/ai/llm/_anthropic.py - Gemini client
Agent/ai/llm/_gemini.py
UI perception
- Mobile connector
Agent/platforms/_mobileconnector.py - UI collectors
Agent/platforms/collectors - Element filters
Agent/platforms/filters/pipeline.py - UI grounding
Agent/platforms/grounding - Visual parsing interface
Agent/ai/vlm/interface.py - OmniParser client
Agent/ai/vlm/_client.py - OmniParser result parser
Agent/ai/vlm/_parser.py
Tools and platforms
- Tool registry
Agent/tools/registry.py - Tool interface
Agent/tools/base.py - Mobile actions
Agent/tools/mobile - Screen answers and checks
Agent/tools/screen
Configuration and usage
- Runtime configuration
Agent/config/config.py - Model configuration
Agent/config/model_config.py - LLM cost tracking
Agent/utilities/_costtracker.py - Model price lookup
Agent/utilities/_pricefetcher.py - Image uploader
Agent/utilities/imguploader/imghandler.py
Other components
- Robot Framework
- OmniParser service
Connections 27
- Robot Framework → Agent keywords: invokes keywords
- Agent keywords → Agent engine: delegates actions
Agent/AgentKeywords.py - Agent engine → Prompt composer: builds prompts
Agent/agent_engine.py - Agent engine → LLM facade: requests completions
Agent/agent_engine.py - Agent engine → Tool registry: selects tools
Agent/agent_engine.py - Agent engine → Screen answers and checks: registers screen tools
Agent/agent_engine.py - Agent engine → Mobile actions: registers mobile tools
Agent/agent_engine.py - LLM facade → Provider factory: creates clients
Agent/ai/llm/facade.py - Provider factory → OpenAI client: selects provider
- Provider factory → Anthropic client: selects provider
- Provider factory → Gemini client: selects provider
- Tool registry → Tool interface: stores tools
Agent/tools/registry.py - Mobile actions → Tool interface: implements interface
Agent/tools/mobile/click_element.py - Screen answers and checks → Tool interface: implements interface
Agent/tools/screen/answer_text.py - Agent engine → Mobile connector: creates platform
Agent/agent_engine.py - Mobile connector → UI collectors: parses UI
Agent/platforms/_mobileconnector.py - Mobile connector → Element filters: filters candidates
Agent/platforms/_mobileconnector.py - Mobile connector → UI grounding: serializes UI context
Agent/platforms/_mobileconnector.py - Agent engine → Visual parsing interface: requests visual parsing
Agent/agent_engine.py - Visual parsing interface → OmniParser client: calls parser service
Agent/ai/vlm/interface.py - Visual parsing interface → OmniParser result parser: parses response
Agent/ai/vlm/interface.py - OmniParser client → OmniParser service: sends image request
- Runtime configuration → Model configuration: loads model defaults
Agent/config/config.py - Model configuration → Model price lookup: initializes price lookup
Agent/config/model_config.py - OpenAI client → LLM cost tracking: records usage cost
Agent/ai/llm/_openaiclient.py - Agent engine → Image uploader: uploads images
Agent/agent_engine.py - Image uploader → Runtime configuration: reads upload settings
Agent/utilities/imguploader/imghandler.py