tahsin2155/handoff

How tahsin2155/handoff works

Handoff is a desktop-operating AI agent. A user enters a task through a dialog; the main loop sends the task and a screenshot to Gemini, dispatches returned actions through a safety gate to desktop controls, then sends a fresh screenshot and feedback for the next turn. The loop also supports user hand-back, configurable model settings, and per-run conversation logs. The graph follows the documented workflow and sampled calls; unsampled loop details are not inferred beyond README statements.

Main entry points: main.py Main Loop · hands.py Mouse and Keyboard · dialogs.py Dialogs · tools.py Tool Definitions

Components 13

User Interaction

Dialogs executions/dialogs.py

Run Orchestration

Main Loop main.py
Model Settings config.py
Run Logs convo_history
Action Dispatch main.py

Model Guidance

System Instructions executions/prompts.py
Tool Definitions executions/tools.py

Desktop Execution

Screen Capture executions/vision.py
Mouse and Keyboard executions/hands.py
Action Results executions/results.py

Other components

User
Gemini Interactions API
Desktop

Connections 17

User → Dialogs: enters task README.md
Dialogs → Main Loop: provides task main.py
Main Loop → Model Settings: reads settings main.py
Main Loop → Screen Capture: captures screen README.md
Main Loop → System Instructions: sets instructions main.py
Main Loop → Tool Definitions: sends tools main.py
Main Loop → Gemini Interactions API: sends request main.py
Gemini Interactions API → Main Loop: returns steps README.md
Main Loop → Action Dispatch: runs turn main.py
Action Dispatch → Mouse and Keyboard: calls actions main.py
Mouse and Keyboard → Desktop: controls screen executions/hands.py
Mouse and Keyboard → Dialogs: requests approval executions/hands.py
Action Dispatch → Tool Definitions: calls hand-back main.py
Tool Definitions → Dialogs: asks for reply executions/tools.py
Main Loop → Action Results: checks outcomes main.py
Mouse and Keyboard → Action Results: returns outcomes executions/hands.py
Main Loop → Run Logs: writes logs README.md

Drawn by GitDiagram on November 15, 2026 from the repository's file tree, README and source files. Read as Markdown ·