How alen-saji-05/androidagent works
This repository is an early Android UI agent: a user goal is decomposed into subgoals, grounded against the visible screen, translated into ADB actions, and reassessed as the screen changes. The implemented screen path combines icon detection and OCR into classified UI elements; planning and grounding support replaceable model-backed and offline implementations. The README describes the end-to-end loop, while sampled code confirms its injected components and internal calls. The verifier, accessibility-tree path, shared memory, and Android app are documented as not yet implemented; concrete CLI wiring is not inferred here.
Main entry points: loop.py Agent loop · pipeline.py Screen parser · planner.py Plan state machine · grounder.py Grounder backends
Components 19
Agent control
- Agent loop
executor/loop.py - Confirmation
executor/loop.py
Planning
- Plan state machine
planner/planner.py - Planner backend
planner/backend.py - Screen digest
planner/digest.py
Screen perception
- Screen parser
perception/pipeline.py - Icon detector
perception/detector.py - OCR backends
perception/ocr.py - Element fusion
perception/fusion.py - Element classifier
perception/classify.py - Screen state
perception/schema.py
Action grounding
- Grounder backends
executor/grounder.py - Grounding heuristics
executor/grounding.py
Device control
- Action executor
executor/executor.py - ADB interface
executor/adb.py
Other components
- User
- Gemini API
- Groq API
- Android device
Connections 19
- User → Agent loop: submits goal
- Agent loop → Plan state machine: decomposes and reassesses
- Plan state machine → Planner backend: requests plan updates
- Planner backend → Gemini API: generates JSON
- Plan state machine → Screen digest: digests screen
- Agent loop → Screen parser: requests screen state
- Screen parser → Icon detector: detects regions
- Screen parser → OCR backends: reads text
- Screen parser → Element fusion: merges detections and text
- Screen parser → Element classifier: classifies elements
- Screen parser → Screen state: builds screen state
- Agent loop → Grounder backends: grounds subgoal
- Grounder backends → Grounding heuristics: uses keyword matching
- Grounder backends → Groq API: requests action decision
- Agent loop → Confirmation: requests approval
- Agent loop → Action executor: dispatches actions
- Action executor → ADB interface: issues device commands
- ADB interface → Android device: taps, types, captures
- Android device → ADB interface: returns screenshot