How tomertir/ai-control-agent works
AI Control Agent turns desktop commands into local OS actions. The principal workflow is text or recognized speech → Claude tool use → local tool dispatch → action result; voice mode repeats this loop. The README also advertises gesture input, but the available source excerpts do not show its implementation, so that branch is marked unverified. Components share agent.py because the repository has a single runtime module; edges are limited to relationships shown in the excerpts or README.
Main entry points: agent.py Tool dispatch
Components 17
Command input
- Mode menu
agent.py - Voice capture
agent.py - Speech recognition
agent.py - Text command
agent.py - Gesture input
agent.py
Agent orchestration
- Agent loop
agent.py - Tool dispatch
agent.py
Desktop actions
- Open application
agent.py - Media control
agent.py - Page scrolling
agent.py - Web search
agent.py - Text entry
agent.py - Screenshot capture
agent.py
External systems
- Claude tool use
- Desktop OS
- Web browser
Other components
- User
Connections 23
- User → Mode menu: selects mode
README.md - Mode menu → Voice capture: starts voice mode
agent.py - Mode menu → Text command: accepts command
README.md - Voice capture → Speech recognition: captures speech
README.md - Speech recognition → Agent loop: supplies text
README.md - Text command → Agent loop: supplies command
README.md - User → Gesture input: provides gestures
- Gesture input → Agent loop: supplies intent
agent.py - Agent loop → Claude tool use: sends command
agent.py - Claude tool use → Agent loop: returns tool choice
agent.py - Agent loop → Tool dispatch: dispatches tool calls
agent.py - Tool dispatch → Open application: invokes
agent.py - Tool dispatch → Media control: invokes
agent.py - Tool dispatch → Page scrolling: invokes
agent.py - Tool dispatch → Web search: invokes
agent.py - Tool dispatch → Text entry: invokes
agent.py - Tool dispatch → Screenshot capture: invokes
agent.py - Open application → Desktop OS: launches application
agent.py - Media control → Desktop OS: sends media keys
agent.py - Page scrolling → Desktop OS: scrolls window
agent.py - Web search → Web browser: opens search
- Text entry → Desktop OS: types text
agent.py - Screenshot capture → Desktop OS: captures screen
agent.py