# seerray-lab/xiaomi-gui-0 architecture

> Architecture diagram of the GitHub repository seerray-lab/xiaomi-gui-0, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/seerray-lab/xiaomi-gui-0
- Repository: https://github.com/seerray-lab/xiaomi-gui-0

## Overview

This repository combines a desktop real-phone GUI agent with two evaluation systems: AndroidWorld’s screenshot-based emulator benchmark and RealMobile’s trajectory-scoring benchmark. The principal interactive workflow starts when a user submits a task in the desktop client; it checks device readiness, runs an agent episode using model inference and device control, then presents and saves the results. AndroidWorld separately runs registered tasks through an agent and environment, while RealMobile scores recorded trajectories against UI hierarchy and task rules. The tree and README document these capabilities; unsampled subsystem wiring is shown only where documented.

Main entry points: [`realmobile/rules/main_eval.py`](https://github.com/seerray-lab/xiaomi-gui-0/blob/main/realmobile/rules/main_eval.py) (Trajectory Evaluator), [`android_world_eval/android_world/episode_runner.py`](https://github.com/seerray-lab/xiaomi-gui-0/blob/main/android_world_eval/android_world/episode_runner.py) (Episode Runner), [`guiness/model/inference_client.py`](https://github.com/seerray-lab/xiaomi-gui-0/blob/main/guiness/model/inference_client.py) (Model Client), [`guiness/gui/app.py`](https://github.com/seerray-lab/xiaomi-gui-0/blob/main/guiness/gui/app.py) (Desktop UI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_desktop["Desktop Agent"]
  node_desktop_app["Desktop UI<br/>[app.py]"]
  node_preflight["Preflight"]
  node_episode_worker["Episode Worker<br/>[episode_worker.py]"]
  node_episode_runner["Episode Runner<br/>[episode_runner.py]"]
  node_conversation[("Conversation State<br/>[chat_manager.py]")]
  node_inference["Model Client"]
  node_model_adapters["Model Adapters"]
end

subgraph group_device["Device Control"]
  node_action_executor["Action Executor<br/>[action_executor.py]"]
  node_device_controller["Device Controller<br/>[adb_controller.py]"]
  node_android_companion["Android Companion<br/>[Routes.kt]"]
  node_accessibility["Accessibility Input"]
end

subgraph group_androidworld["AndroidWorld Benchmark"]
  node_aw_registry["Task Registry<br/>[registry.py]"]
  node_aw_episode["Episode Runner<br/>[episode_runner.py]"]
  node_aw_agent["Screenshot Agent"]
  node_aw_environment["Android Environment"]
  node_aw_tasks["Task Evaluators"]
end

subgraph group_realmobile["RealMobile Scoring"]
  node_real_evaluator["Trajectory Evaluator<br/>[main_eval.py]"]
  node_xpath_rules["XPath Rules<br/>[evaluator_xpath.py]"]
  node_rule_modules["Task Rules"]
  node_score["Subgoal Scores"]
end

node_user(("Task User"))
node_model_service{{"Model API"}}
node_phone(("Android Phone"))
node_aw_model{{"Vision Model API"}}
node_real_trajectory["Recorded Trajectory"]
node_ui_hierarchy["UI Hierarchy XML"]
node_ocr{{"PaddleOCR"}}

node_user -->|"submits task"| node_desktop_app
node_desktop_app -->|"checks readiness"| node_preflight
node_desktop_app -->|"starts episode"| node_episode_worker
node_episode_worker -->|"runs task"| node_episode_runner
node_episode_worker -->|"updates state"| node_conversation
node_episode_runner -->|"requests actions"| node_inference
node_inference -->|"selects and parses"| node_model_adapters
node_inference -->|"sends prompts"| node_model_service
node_model_service -->|"returns response"| node_inference
node_episode_runner -->|"executes actions"| node_action_executor
node_action_executor -->|"controls device"| node_device_controller
node_device_controller -->|"ADB operations"| node_phone
node_android_companion -->|"dispatches input"| node_accessibility
node_accessibility -->|"injects gestures"| node_phone
node_aw_registry -->|"provides tasks"| node_aw_episode
node_aw_episode -->|"requests decisions"| node_aw_agent
node_aw_agent -->|"sends screenshot prompts"| node_aw_model
node_aw_episode -->|"steps environment"| node_aw_environment
node_aw_episode -->|"evaluates task"| node_aw_tasks
node_real_trajectory -->|"supplies rollout"| node_real_evaluator
node_real_evaluator -->|"checks hierarchy"| node_xpath_rules
node_real_evaluator -->|"applies task rules"| node_rule_modules
node_xpath_rules -->|"reads XML"| node_ui_hierarchy
node_real_evaluator -.->|"uses text recognition"| node_ocr
node_real_evaluator -->|"produces subgoal scores"| node_score

click node_desktop_app "https://github.com/seerray-lab/xiaomi-gui-0/blob/main/guiness/gui/app.py"
click node_preflight "https://github.com/seerray-lab/xiaomi-gui-0/blob/main/guiness/gui/workers/preflight_worker.py"
click node_episode_worker "https://github.com/seerray-lab/xiaomi-gui-0/blob/main/guiness/gui/workers/episode_worker.py"
click node_episode_runner "https://github.com/seerray-lab/xiaomi-gui-0/blob/main/guiness/runner/episode_runner.py"
click node_conversation "https://github.com/seerray-lab/xiaomi-gui-0/blob/main/guiness/gui/chat_manager.py"
click node_inference "https://github.com/seerray-lab/xiaomi-gui-0/blob/main/guiness/model/inference_client.py"
click node_model_adapters "https://github.com/seerray-lab/xiaomi-gui-0/tree/main/guiness/model/adapters"
click node_action_executor "https://github.com/seerray-lab/xiaomi-gui-0/blob/main/guiness/action/action_executor.py"
click node_device_controller "https://github.com/seerray-lab/xiaomi-gui-0/blob/main/guiness/device/adb_controller.py"
click node_android_companion "https://github.com/seerray-lab/xiaomi-gui-0/blob/main/guiness/android/app/src/main/java/com/guiclaw/controller/server/Routes.kt"
click node_accessibility "https://github.com/seerray-lab/xiaomi-gui-0/blob/main/guiness/android/app/src/main/java/com/guiclaw/controller/service/ControlAccessibilityService.kt"
click node_aw_registry "https://github.com/seerray-lab/xiaomi-gui-0/blob/main/android_world_eval/android_world/registry.py"
click node_aw_episode "https://github.com/seerray-lab/xiaomi-gui-0/blob/main/android_world_eval/android_world/episode_runner.py"
click node_aw_agent "https://github.com/seerray-lab/xiaomi-gui-0/tree/main/android_world_eval/android_world/agents/cv_agent"
click node_aw_environment "https://github.com/seerray-lab/xiaomi-gui-0/blob/main/android_world_eval/android_world/env/android_world_controller.py"
click node_aw_tasks "https://github.com/seerray-lab/xiaomi-gui-0/tree/main/android_world_eval/android_world/task_evals"
click node_real_evaluator "https://github.com/seerray-lab/xiaomi-gui-0/blob/main/realmobile/rules/main_eval.py"
click node_xpath_rules "https://github.com/seerray-lab/xiaomi-gui-0/blob/main/realmobile/rules/evaluator_xpath.py"
click node_rule_modules "https://github.com/seerray-lab/xiaomi-gui-0/tree/main/realmobile/rules"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_desktop_app,node_preflight,node_episode_worker,node_episode_runner,node_conversation,node_inference,node_model_adapters,node_ui_hierarchy toneBlue
class node_action_executor,node_device_controller,node_android_companion,node_accessibility toneAmber
class node_aw_registry,node_aw_episode,node_aw_agent,node_aw_environment,node_aw_tasks,node_model_service,node_aw_model toneMint
class node_real_evaluator,node_xpath_rules,node_rule_modules,node_score,node_user toneRose
class node_phone,node_real_trajectory,node_ocr toneIndigo
```

## Components

### Desktop Agent

- [Desktop UI](https://github.com/seerray-lab/xiaomi-gui-0/blob/main/guiness/gui/app.py): `guiness/gui/app.py`
- [Preflight](https://github.com/seerray-lab/xiaomi-gui-0/blob/main/guiness/gui/workers/preflight_worker.py): `guiness/gui/workers/preflight_worker.py`
- [Episode Worker](https://github.com/seerray-lab/xiaomi-gui-0/blob/main/guiness/gui/workers/episode_worker.py): `guiness/gui/workers/episode_worker.py`
- [Episode Runner](https://github.com/seerray-lab/xiaomi-gui-0/blob/main/guiness/runner/episode_runner.py): `guiness/runner/episode_runner.py`
- [Conversation State](https://github.com/seerray-lab/xiaomi-gui-0/blob/main/guiness/gui/chat_manager.py): `guiness/gui/chat_manager.py`
- [Model Client](https://github.com/seerray-lab/xiaomi-gui-0/blob/main/guiness/model/inference_client.py): `guiness/model/inference_client.py`
- [Model Adapters](https://github.com/seerray-lab/xiaomi-gui-0/tree/main/guiness/model/adapters): `guiness/model/adapters`

### Device Control

- [Action Executor](https://github.com/seerray-lab/xiaomi-gui-0/blob/main/guiness/action/action_executor.py): `guiness/action/action_executor.py`
- [Device Controller](https://github.com/seerray-lab/xiaomi-gui-0/blob/main/guiness/device/adb_controller.py): `guiness/device/adb_controller.py`
- [Android Companion](https://github.com/seerray-lab/xiaomi-gui-0/blob/main/guiness/android/app/src/main/java/com/guiclaw/controller/server/Routes.kt): `guiness/android/app/src/main/java/com/guiclaw/controller/server/Routes.kt`
- [Accessibility Input](https://github.com/seerray-lab/xiaomi-gui-0/blob/main/guiness/android/app/src/main/java/com/guiclaw/controller/service/ControlAccessibilityService.kt): `guiness/android/app/src/main/java/com/guiclaw/controller/service/ControlAccessibilityService.kt`

### AndroidWorld Benchmark

- [Task Registry](https://github.com/seerray-lab/xiaomi-gui-0/blob/main/android_world_eval/android_world/registry.py): `android_world_eval/android_world/registry.py`
- [Episode Runner](https://github.com/seerray-lab/xiaomi-gui-0/blob/main/android_world_eval/android_world/episode_runner.py): `android_world_eval/android_world/episode_runner.py`
- [Screenshot Agent](https://github.com/seerray-lab/xiaomi-gui-0/tree/main/android_world_eval/android_world/agents/cv_agent): `android_world_eval/android_world/agents/cv_agent`
- [Android Environment](https://github.com/seerray-lab/xiaomi-gui-0/blob/main/android_world_eval/android_world/env/android_world_controller.py): `android_world_eval/android_world/env/android_world_controller.py`
- [Task Evaluators](https://github.com/seerray-lab/xiaomi-gui-0/tree/main/android_world_eval/android_world/task_evals): `android_world_eval/android_world/task_evals`

### RealMobile Scoring

- [Trajectory Evaluator](https://github.com/seerray-lab/xiaomi-gui-0/blob/main/realmobile/rules/main_eval.py): `realmobile/rules/main_eval.py`
- [XPath Rules](https://github.com/seerray-lab/xiaomi-gui-0/blob/main/realmobile/rules/evaluator_xpath.py): `realmobile/rules/evaluator_xpath.py`
- [Task Rules](https://github.com/seerray-lab/xiaomi-gui-0/tree/main/realmobile/rules): `realmobile/rules`
- **Subgoal Scores**

### Other components

- **Task User**
- **Model API**
- **Android Phone**
- **Vision Model API**
- **Recorded Trajectory**
- **UI Hierarchy XML**
- **PaddleOCR**

## Connections

- Task User → Desktop UI: submits task
- Desktop UI → Preflight: checks readiness
- Desktop UI → Episode Worker: starts episode
- Episode Worker → Episode Runner: runs task
- Episode Worker → Conversation State: updates state
- Episode Runner → Model Client: requests actions
- Model Client → Model Adapters: selects and parses
- Model Client → Model API: sends prompts
- Model API → Model Client: returns response
- Episode Runner → Action Executor: executes actions
- Action Executor → Device Controller: controls device
- Device Controller → Android Phone: ADB operations
- Android Companion → Accessibility Input: dispatches input
- Accessibility Input → Android Phone: injects gestures
- Task Registry → Episode Runner: provides tasks
- Episode Runner → Screenshot Agent: requests decisions
- Screenshot Agent → Vision Model API: sends screenshot prompts
- Episode Runner → Android Environment: steps environment
- Episode Runner → Task Evaluators: evaluates task
- Recorded Trajectory → Trajectory Evaluator: supplies rollout
- Trajectory Evaluator → XPath Rules: checks hierarchy
- Trajectory Evaluator → Task Rules: applies task rules
- XPath Rules → UI Hierarchy XML: reads XML
- Trajectory Evaluator → PaddleOCR: uses text recognition
- Trajectory Evaluator → Subgoal Scores: produces subgoal scores