# mobilellm/agentprog architecture

> Architecture diagram of the GitHub repository mobilellm/agentprog, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mobilellm/agentprog
- Repository: https://github.com/mobilellm/agentprog

## Overview

AgentProg is a long-horizon Android GUI agent: a task description is turned into a Semantic Task Program, then interpreted step-by-step into device actions using live screenshots and UI observations. Its workflow context and belief state support control-flow pruning, persistent task data, repeated-step recall and recovery. AndroidWorld is the companion evaluation environment, with registered task suites, device interaction and task-specific success checks. Edges below are limited to relationships explicit in the README or visible in the supplied excerpts/SOURCE INDEX; paths for unsampled subsystems identify only existing directories.

Main entry points: [`agentprog/plan/code_exec/workflow/core/agentprog.py`](https://github.com/mobilellm/agentprog/blob/main/agentprog/plan/code_exec/workflow/core/agentprog.py) (Execution controller), [`agentprog/plan/code_exec/workflow/workflow_query_llm.py`](https://github.com/mobilellm/agentprog/blob/main/agentprog/plan/code_exec/workflow/workflow_query_llm.py) (Program runner), [`eval/android_world/run.py`](https://github.com/mobilellm/agentprog/blob/main/eval/android_world/run.py) (Evaluation runner), [`agentprog/plan/code_exec/workflow/pipeline.py`](https://github.com/mobilellm/agentprog/blob/main/agentprog/plan/code_exec/workflow/pipeline.py) (Workflow pipeline)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_agent["AgentProg runtime"]
  node_cli["CLI entry<br/>[cli.py]"]
  node_pipeline["Workflow pipeline<br/>[pipeline.py]"]
  node_fm["Foundation model<br/>[fm.py]"]
  node_llm["Model API utilities<br/>[general_utils.py]"]
end

subgraph group_planning["Program and context"]
  node_generator["Program generation"]
  node_runner["Program runner"]
  node_core["Execution controller<br/>[agentprog.py]"]
  node_context["Workflow context<br/>[workflow_utils.py]"]
  node_context_view["Agent context<br/>[agentprog_utils.py]"]
  node_belief["Belief state<br/>[belief_state.py]"]
  node_prompts["Program prompts"]
end

subgraph group_device["Android interaction"]
  node_mobile["Mobile API<br/>[mobile_utils.py]"]
  node_locator["UI-TARS locator<br/>[ui_tars_utils.py]"]
end

subgraph group_benchmark["AndroidWorld evaluation"]
  node_eval_runner["Evaluation runner<br/>[run.py]"]
  node_registry["Task registry<br/>[registry.py]"]
  node_task_eval["Task evaluation<br/>[task_eval.py]"]
  node_retrieval["Information retrieval tasks"]
  node_env["Async environment<br/>[interface.py]"]
  node_controller["World controller"]
  node_adb["ADB utilities<br/>[adb_utils.py]"]
  node_ui_repr["UI representation"]
  node_app_checks["Retrieval task helpers"]
  node_agentprog["AgentProg adapter<br/>[agentprog.py]"]
end

node_user(("Task user"))
node_android(("Android device"))

node_user -->|"submits task"| node_cli
node_cli -->|"starts pipeline"| node_pipeline
node_pipeline -->|"generates program"| node_generator
node_pipeline -->|"runs program"| node_runner
node_runner -->|"dispatches execution"| node_core
node_core -->|"updates context"| node_context
node_core -->|"uses belief state"| node_belief
node_core -->|"builds agent context"| node_context_view
node_core -->|"uses prompt set"| node_prompts
node_runner -->|"captures screenshots"| node_mobile
node_runner -->|"queries model"| node_fm
node_fm -->|"formats model requests"| node_llm
node_mobile -->|"selects locator"| node_locator
node_mobile -.->|"acts on device"| node_android
node_eval_runner -->|"loads tasks"| node_registry
node_registry -->|"registers evaluations"| node_task_eval
node_registry -->|"registers tasks"| node_retrieval
node_retrieval -->|"extends evaluation"| node_task_eval
node_retrieval -->|"sets task state"| node_app_checks
node_env -->|"uses controller"| node_controller
node_controller -->|"converts UI tree"| node_ui_repr
node_env -->|"dispatches actions"| node_adb
node_eval_runner -.->|"runs against environment"| node_env
node_eval_runner -->|"selects AgentProg agent"| node_agentprog

click node_cli "https://github.com/mobilellm/agentprog/blob/main/agentprog/cli.py"
click node_pipeline "https://github.com/mobilellm/agentprog/blob/main/agentprog/plan/code_exec/workflow/pipeline.py"
click node_generator "https://github.com/mobilellm/agentprog/blob/main/agentprog/plan/code_exec/workflow/generate_workflow.py"
click node_runner "https://github.com/mobilellm/agentprog/blob/main/agentprog/plan/code_exec/workflow/workflow_query_llm.py"
click node_core "https://github.com/mobilellm/agentprog/blob/main/agentprog/plan/code_exec/workflow/core/agentprog.py"
click node_context "https://github.com/mobilellm/agentprog/blob/main/agentprog/plan/workflow_utils.py"
click node_context_view "https://github.com/mobilellm/agentprog/blob/main/agentprog/plan/agentprog_utils.py"
click node_belief "https://github.com/mobilellm/agentprog/blob/main/agentprog/plan/belief/belief_state.py"
click node_prompts "https://github.com/mobilellm/agentprog/tree/main/agentprog/plan/code_exec/workflow/workflow_prompts"
click node_fm "https://github.com/mobilellm/agentprog/blob/main/agentprog/all_utils/fm.py"
click node_llm "https://github.com/mobilellm/agentprog/blob/main/agentprog/all_utils/general_utils.py"
click node_mobile "https://github.com/mobilellm/agentprog/blob/main/agentprog/all_utils/mobile_utils.py"
click node_locator "https://github.com/mobilellm/agentprog/blob/main/agentprog/all_utils/ui_tars_utils.py"
click node_eval_runner "https://github.com/mobilellm/agentprog/blob/main/eval/android_world/run.py"
click node_registry "https://github.com/mobilellm/agentprog/blob/main/eval/android_world/android_world/registry.py"
click node_task_eval "https://github.com/mobilellm/agentprog/blob/main/eval/android_world/android_world/task_evals/task_eval.py"
click node_retrieval "https://github.com/mobilellm/agentprog/blob/main/eval/android_world/android_world/task_evals/information_retrieval/information_retrieval.py"
click node_env "https://github.com/mobilellm/agentprog/blob/main/eval/android_world/android_world/env/interface.py"
click node_controller "https://github.com/mobilellm/agentprog/blob/main/eval/android_world/android_world/env/android_world_controller.py"
click node_adb "https://github.com/mobilellm/agentprog/blob/main/eval/android_world/android_world/env/adb_utils.py"
click node_ui_repr "https://github.com/mobilellm/agentprog/blob/main/eval/android_world/android_world/env/representation_utils.py"
click node_app_checks "https://github.com/mobilellm/agentprog/tree/main/eval/android_world/android_world/task_evals/information_retrieval"
click node_agentprog "https://github.com/mobilellm/agentprog/blob/main/eval/android_world/android_world/agents/agentprog.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_pipeline,node_fm,node_llm toneBlue
class node_generator,node_runner,node_core,node_context,node_context_view,node_belief,node_prompts toneAmber
class node_mobile,node_locator toneMint
class node_eval_runner,node_registry,node_task_eval,node_retrieval,node_env,node_controller,node_adb,node_ui_repr,node_app_checks,node_agentprog,node_user toneRose
class node_android toneIndigo
```

## Components

### AgentProg runtime

- [CLI entry](https://github.com/mobilellm/agentprog/blob/main/agentprog/cli.py): `agentprog/cli.py`
- [Workflow pipeline](https://github.com/mobilellm/agentprog/blob/main/agentprog/plan/code_exec/workflow/pipeline.py): `agentprog/plan/code_exec/workflow/pipeline.py`
- [Foundation model](https://github.com/mobilellm/agentprog/blob/main/agentprog/all_utils/fm.py): `agentprog/all_utils/fm.py`
- [Model API utilities](https://github.com/mobilellm/agentprog/blob/main/agentprog/all_utils/general_utils.py): `agentprog/all_utils/general_utils.py`

### Program and context

- [Program generation](https://github.com/mobilellm/agentprog/blob/main/agentprog/plan/code_exec/workflow/generate_workflow.py): `agentprog/plan/code_exec/workflow/generate_workflow.py`
- [Program runner](https://github.com/mobilellm/agentprog/blob/main/agentprog/plan/code_exec/workflow/workflow_query_llm.py): `agentprog/plan/code_exec/workflow/workflow_query_llm.py`
- [Execution controller](https://github.com/mobilellm/agentprog/blob/main/agentprog/plan/code_exec/workflow/core/agentprog.py): `agentprog/plan/code_exec/workflow/core/agentprog.py`
- [Workflow context](https://github.com/mobilellm/agentprog/blob/main/agentprog/plan/workflow_utils.py): `agentprog/plan/workflow_utils.py`
- [Agent context](https://github.com/mobilellm/agentprog/blob/main/agentprog/plan/agentprog_utils.py): `agentprog/plan/agentprog_utils.py`
- [Belief state](https://github.com/mobilellm/agentprog/blob/main/agentprog/plan/belief/belief_state.py): `agentprog/plan/belief/belief_state.py`
- [Program prompts](https://github.com/mobilellm/agentprog/tree/main/agentprog/plan/code_exec/workflow/workflow_prompts): `agentprog/plan/code_exec/workflow/workflow_prompts`

### Android interaction

- [Mobile API](https://github.com/mobilellm/agentprog/blob/main/agentprog/all_utils/mobile_utils.py): `agentprog/all_utils/mobile_utils.py`
- [UI-TARS locator](https://github.com/mobilellm/agentprog/blob/main/agentprog/all_utils/ui_tars_utils.py): `agentprog/all_utils/ui_tars_utils.py`

### AndroidWorld evaluation

- [Evaluation runner](https://github.com/mobilellm/agentprog/blob/main/eval/android_world/run.py): `eval/android_world/run.py`
- [Task registry](https://github.com/mobilellm/agentprog/blob/main/eval/android_world/android_world/registry.py): `eval/android_world/android_world/registry.py`
- [Task evaluation](https://github.com/mobilellm/agentprog/blob/main/eval/android_world/android_world/task_evals/task_eval.py): `eval/android_world/android_world/task_evals/task_eval.py`
- [Information retrieval tasks](https://github.com/mobilellm/agentprog/blob/main/eval/android_world/android_world/task_evals/information_retrieval/information_retrieval.py): `eval/android_world/android_world/task_evals/information_retrieval/information_retrieval.py`
- [Async environment](https://github.com/mobilellm/agentprog/blob/main/eval/android_world/android_world/env/interface.py): `eval/android_world/android_world/env/interface.py`
- [World controller](https://github.com/mobilellm/agentprog/blob/main/eval/android_world/android_world/env/android_world_controller.py): `eval/android_world/android_world/env/android_world_controller.py`
- [ADB utilities](https://github.com/mobilellm/agentprog/blob/main/eval/android_world/android_world/env/adb_utils.py): `eval/android_world/android_world/env/adb_utils.py`
- [UI representation](https://github.com/mobilellm/agentprog/blob/main/eval/android_world/android_world/env/representation_utils.py): `eval/android_world/android_world/env/representation_utils.py`
- [Retrieval task helpers](https://github.com/mobilellm/agentprog/tree/main/eval/android_world/android_world/task_evals/information_retrieval): `eval/android_world/android_world/task_evals/information_retrieval`
- [AgentProg adapter](https://github.com/mobilellm/agentprog/blob/main/eval/android_world/android_world/agents/agentprog.py): `eval/android_world/android_world/agents/agentprog.py`

### Other components

- **Task user**
- **Android device**

## Connections

- Task user → CLI entry: submits task (evidence: `README.md`)
- CLI entry → Workflow pipeline: starts pipeline (evidence: `agentprog/cli.py`)
- Workflow pipeline → Program generation: generates program (evidence: `agentprog/plan/code_exec/workflow/pipeline.py`)
- Workflow pipeline → Program runner: runs program (evidence: `agentprog/plan/code_exec/workflow/pipeline.py`)
- Program runner → Execution controller: dispatches execution (evidence: `agentprog/plan/code_exec/workflow/workflow_query_llm.py`)
- Execution controller → Workflow context: updates context (evidence: `agentprog/plan/code_exec/workflow/core/agentprog.py`)
- Execution controller → Belief state: uses belief state (evidence: `agentprog/plan/code_exec/workflow/core/agentprog.py`)
- Execution controller → Agent context: builds agent context (evidence: `agentprog/plan/code_exec/workflow/core/agentprog.py`)
- Execution controller → Program prompts: uses prompt set (evidence: `agentprog/plan/code_exec/workflow/core/agentprog.py`)
- Program runner → Mobile API: captures screenshots (evidence: `agentprog/plan/code_exec/workflow/workflow_query_llm.py`)
- Program runner → Foundation model: queries model (evidence: `agentprog/plan/code_exec/workflow/workflow_query_llm.py`)
- Foundation model → Model API utilities: formats model requests (evidence: `agentprog/all_utils/fm.py`)
- Mobile API → UI-TARS locator: selects locator (evidence: `agentprog/all_utils/mobile_utils.py`)
- Mobile API → Android device: acts on device
- Evaluation runner → Task registry: loads tasks (evidence: `eval/android_world/run.py`)
- Task registry → Task evaluation: registers evaluations (evidence: `eval/android_world/android_world/registry.py`)
- Task registry → Information retrieval tasks: registers tasks (evidence: `eval/android_world/android_world/registry.py`)
- Information retrieval tasks → Task evaluation: extends evaluation (evidence: `eval/android_world/android_world/task_evals/information_retrieval/information_retrieval.py`)
- Information retrieval tasks → Retrieval task helpers: sets task state (evidence: `eval/android_world/android_world/task_evals/information_retrieval/information_retrieval.py`)
- Async environment → World controller: uses controller (evidence: `eval/android_world/android_world/env/interface.py`)
- World controller → UI representation: converts UI tree (evidence: `eval/android_world/android_world/env/android_world_controller.py`)
- Async environment → ADB utilities: dispatches actions (evidence: `eval/android_world/android_world/env/interface.py`)
- Evaluation runner → Async environment: runs against environment (evidence: `eval/android_world/run.py`)
- Evaluation runner → AgentProg adapter: selects AgentProg agent (evidence: `eval/android_world/run.py`)
