How dinesh210805/aura-android-use works
AURA is an Android assistant that lets a person or an external MCP client operate real phone apps. The principal workflow is request intake (voice, text, or live conversation), agent planning with a selected LLM, guarded tool execution, screen reading and accessibility actions, then verification and response. The app also exposes a LAN MCP server, with an optional desktop connector, and includes persistent memory and skill support. Relationships below follow the README’s documented architecture; unsampled implementation details are not inferred.
Main entry points: AuraAgent.kt On-device agent · AuraAccessibilityService.kt Accessibility service · McpServerController.kt Phone MCP server · ActionGuard.kt Tool safety gate
Components 20
Phone experience
- App entry
aura-android/app/src/main/java/com/aura/aura_ui/MainActivity.kt - Conversation UI
aura-android/app/src/main/java/com/aura/aura_ui/conversation/ConversationViewModel.kt - Voice capture
aura-android/app/src/main/java/com/aura/aura_ui/voice/VoiceCaptureController.kt - Live conversation
aura-android/app/src/main/java/com/aura/aura_ui/agent/conversation/CompanionLiveController.kt - Assistant overlay
aura-android/app/src/main/java/com/aura/aura_ui/overlay/AuraOverlayService.kt
Agent runtime
- On-device agent
aura-android/app/src/main/java/com/aura/aura_ui/agent/AuraAgent.kt - LLM providers
aura-android/app/src/main/java/com/aura/aura_ui/agent/llm/LlmEndpointCatalog.kt - Tool safety gate
aura-android/app/src/main/java/com/aura/aura_ui/agent/mcpbridge/ActionGuard.kt - Agent skills
aura-android/app/src/main/java/com/aura/aura_ui/agent/skills/SkillManager.kt - Completion checks
aura-android/app/src/main/java/com/aura/aura_ui/agent/proof/CompletionGate.kt
Phone interaction
- Accessibility service
aura-android/app/src/main/java/com/aura/aura_ui/accessibility/AuraAccessibilityService.kt - Screen perception
aura-android/app/src/main/java/com/aura/aura_ui/accessibility/UITreeExtractor.kt - Gesture execution
aura-android/app/src/main/java/com/aura/aura_ui/accessibility/gesture/GestureDispatcher.kt
MCP access
- Phone MCP server
aura-android/mcp-server/src/main/kotlin/com/aura/mcp/McpServerController.kt - Phone tool bridges
aura-android/mcp-server/src/main/kotlin/com/aura/mcp/tools/PerceiveScreenTool.kt - Desktop connector
aura-mcp-connect/src/index.js
Context and memory
- Agent memory
aura-android/app/src/main/java/com/aura/aura_ui/agent/memory/MemoryService.kt - Encrypted memory store
aura-android/app/src/main/java/com/aura/aura_ui/agent/memory/EncryptedJsonStore.kt
Other components
- Phone user
- Computer MCP client
Connections 22
- Phone user → Conversation UI: asks by text
- Phone user → Voice capture: speaks request
- Phone user → Live conversation: starts conversation
- Conversation UI → On-device agent: hands off task
- Voice capture → On-device agent: sends transcript
- Live conversation → On-device agent: calls ask_aura
- App entry → Assistant overlay: starts assistant
aura-android/app/src/main/java/com/aura/aura_ui/MainActivity.kt - Assistant overlay → On-device agent: runs task
aura-android/app/src/main/java/com/aura/aura_ui/overlay/AuraOverlayService.kt - On-device agent → LLM providers: requests next step
aura-android/app/src/main/java/com/aura/aura_ui/agent/AuraAgent.kt - On-device agent → Tool safety gate: submits tool call
- On-device agent → Agent skills: uses skills
- Tool safety gate → Accessibility service: permits device action
- Accessibility service → Screen perception: reads screen
aura-android/app/src/main/java/com/aura/aura_ui/accessibility/AuraAccessibilityService.kt - Accessibility service → Gesture execution: dispatches actions
- On-device agent → Completion checks: checks completion
- Computer MCP client → Desktop connector: sends MCP requests
- Desktop connector → Phone MCP server: connects over LAN
- Phone MCP server → Tool safety gate: gates tool calls
- Phone MCP server → Phone tool bridges: dispatches tools
- On-device agent → Agent memory: uses remembered context
- Agent memory → Encrypted memory store: persists data
aura-android/app/src/main/java/com/aura/aura_ui/agent/memory/MemoryService.kt - On-device agent → Phone user: returns result