How midunthangavel/myuniverse works
Synapse is a phone-native personal AI agent with a web cockpit, backend planning and memory services, and Android device automation. The primary workflow is a user request entering through the cockpit or backend API, being planned by the agent, then executed against a phone and checked against the resulting screen state. The README also describes voice, proactive routines, multimodal perception, and memory. Some links below follow explicit README or SOURCE INDEX references; unsampled subsystems are included where documented, but their internal wiring is not inferred.
Main entry points: agent.py Agent Orchestrator · main.js Dashboard UI · main.py API & WebSockets · controller_factory.py Controller Factory
Components 34
Web Cockpit
- Dashboard UI
src/main.js - Agent Interaction
src/agent.js - Voice Interface
src/audio.js - Animated Avatar
src/character.js - Phone Mirror
src/phone.js - Memory Explorer
src/memory.js
Agent Backend
- API & WebSockets
backend/app/main.py
Reasoning & Memory
- Agent Orchestrator
backend/app/agent.py - Fast Classifier
backend/app/fast_classifier.py - LLM Providers
backend/app/llm_factory.py - Vision Perception
backend/app/vision.py - Local VLM
backend/app/vlm_perception.py - Prompt Manager
backend/app/prompt_manager.py - Guardrails
backend/app/guardrails.py - Proactive Routines
backend/app/proactive.py - Memory Services
backend/app/memory.py - App Trajectories
backend/app/app_memory.py - Vector Memory
backend/app/vector_memory.py - Tool Registry
backend/app/tool_registry.py - Task Progressor
backend/app/progressor.py - Action Reflector
backend/app/reflector.py
Phone Automation
- Action Engine
backend/app/device/action_engine.py - State Engine
backend/app/device/state_engine.py - UI Grounding
backend/app/device/grounding_engine.py - Outcome Verification
backend/app/device/verification_engine.py - UI Parser
backend/app/device/ui_parser.py - Controller Factory
backend/app/device/controller_factory.py - ADB Controller
backend/app/device/android_controller.py - iOS Controller
backend/app/device/ios_controller.py - Simulator Controller
backend/app/device/sim_controller.py
Android Client
- ADB Bridge
android/bridge/adb_bridge.py - Accessibility Service
android/app/src/main/java/com/synapse/ai/accessibility/SynapseAccessibilityService.kt - WebSocket Client
android/app/src/main/java/com/synapse/ai/network/SynapseWebSocketClient.kt
Other components
- User
Connections 36
- User → Dashboard UI: uses
- Dashboard UI → Agent Interaction: dispatches to
src/main.js - Dashboard UI → Voice Interface: wires
src/main.js - Dashboard UI → Animated Avatar: renders
src/main.js - Dashboard UI → Phone Mirror: wires
src/main.js - Dashboard UI → Memory Explorer: wires
src/main.js - Agent Interaction → API & WebSockets: requests
- API & WebSockets → Agent Orchestrator: invokes
backend/app/main.py - API & WebSockets → Proactive Routines: uses
backend/app/main.py - API & WebSockets → App Trajectories: serves
backend/app/main.py - API & WebSockets → Controller Factory: routes device calls
backend/app/main.py - API & WebSockets → LLM Providers: configures
backend/app/main.py - Agent Orchestrator → LLM Providers: plans with
backend/app/agent.py - Agent Orchestrator → Vision Perception: uses
backend/app/agent.py - Agent Orchestrator → Prompt Manager: uses
backend/app/agent.py - Agent Orchestrator → Guardrails: checks
backend/app/agent.py - Agent Orchestrator → Tool Registry: dispatches tools
backend/app/agent.py - Agent Orchestrator → Fast Classifier: uses
backend/app/agent.py - Agent Orchestrator → Memory Services: recalls
backend/app/agent.py - Agent Orchestrator → Vector Memory: uses
backend/app/agent.py - Agent Orchestrator → Task Progressor: tracks tasks
backend/app/agent.py - Agent Orchestrator → Action Reflector: reflects with
backend/app/agent.py - App Trajectories → Vector Memory: stores trajectories
backend/app/app_memory.py - Vision Perception → Local VLM: may use
- Action Engine → State Engine: perceives state
backend/app/device/action_engine.py - Action Engine → UI Grounding: resolves targets
backend/app/device/action_engine.py - Action Engine → Outcome Verification: verifies actions
backend/app/device/action_engine.py - Action Engine → Controller Factory: gets controller
backend/app/device/action_engine.py - State Engine → UI Parser: uses
backend/app/device/state_engine.py - State Engine → ADB Controller: reads hierarchy
backend/app/device/state_engine.py - Outcome Verification → State Engine: compares states
backend/app/device/verification_engine.py - Controller Factory → ADB Controller: selects
backend/app/device/controller_factory.py - Controller Factory → iOS Controller: selects
backend/app/device/controller_factory.py - Controller Factory → Simulator Controller: selects
backend/app/device/controller_factory.py - ADB Controller → ADB Bridge: controls device via
- WebSocket Client → Accessibility Service: uses
android/app/src/main/java/com/synapse/ai/network/SynapseWebSocketClient.kt