How seerray-lab/xiaomi-gui-0 works
This repository combines a desktop real-phone GUI agent with two evaluation systems: AndroidWorld’s screenshot-based emulator benchmark and RealMobile’s trajectory-scoring benchmark. The principal interactive workflow starts when a user submits a task in the desktop client; it checks device readiness, runs an agent episode using model inference and device control, then presents and saves the results. AndroidWorld separately runs registered tasks through an agent and environment, while RealMobile scores recorded trajectories against UI hierarchy and task rules. The tree and README document these capabilities; unsampled subsystem wiring is shown only where documented.
Main entry points: main_eval.py Trajectory Evaluator · episode_runner.py Episode Runner · inference_client.py Model Client · app.py Desktop UI
Components 27
Desktop Agent
- Desktop UI
guiness/gui/app.py - Preflight
guiness/gui/workers/preflight_worker.py - Episode Worker
guiness/gui/workers/episode_worker.py - Episode Runner
guiness/runner/episode_runner.py - Conversation State
guiness/gui/chat_manager.py - Model Client
guiness/model/inference_client.py - Model Adapters
guiness/model/adapters
Device Control
- Action Executor
guiness/action/action_executor.py - Device Controller
guiness/device/adb_controller.py - Android Companion
guiness/android/app/src/main/java/com/guiclaw/controller/server/Routes.kt - Accessibility Input
guiness/android/app/src/main/java/com/guiclaw/controller/service/ControlAccessibilityService.kt
AndroidWorld Benchmark
- Task Registry
android_world_eval/android_world/registry.py - Episode Runner
android_world_eval/android_world/episode_runner.py - Screenshot Agent
android_world_eval/android_world/agents/cv_agent - Android Environment
android_world_eval/android_world/env/android_world_controller.py - Task Evaluators
android_world_eval/android_world/task_evals
RealMobile Scoring
- Trajectory Evaluator
realmobile/rules/main_eval.py - XPath Rules
realmobile/rules/evaluator_xpath.py - Task Rules
realmobile/rules - Subgoal Scores
Other components
- Task User
- Model API
- Android Phone
- Vision Model API
- Recorded Trajectory
- UI Hierarchy XML
- PaddleOCR
Connections 25
- Task User → Desktop UI: submits task
- Desktop UI → Preflight: checks readiness
- Desktop UI → Episode Worker: starts episode
- Episode Worker → Episode Runner: runs task
- Episode Worker → Conversation State: updates state
- Episode Runner → Model Client: requests actions
- Model Client → Model Adapters: selects and parses
- Model Client → Model API: sends prompts
- Model API → Model Client: returns response
- Episode Runner → Action Executor: executes actions
- Action Executor → Device Controller: controls device
- Device Controller → Android Phone: ADB operations
- Android Companion → Accessibility Input: dispatches input
- Accessibility Input → Android Phone: injects gestures
- Task Registry → Episode Runner: provides tasks
- Episode Runner → Screenshot Agent: requests decisions
- Screenshot Agent → Vision Model API: sends screenshot prompts
- Episode Runner → Android Environment: steps environment
- Episode Runner → Task Evaluators: evaluates task
- Recorded Trajectory → Trajectory Evaluator: supplies rollout
- Trajectory Evaluator → XPath Rules: checks hierarchy
- Trajectory Evaluator → Task Rules: applies task rules
- XPath Rules → UI Hierarchy XML: reads XML
- Trajectory Evaluator → PaddleOCR: uses text recognition
- Trajectory Evaluator → Subgoal Scores: produces subgoal scores