How faizan6xyz/agent-for-local-tasks_using_vision_model works
This repository is a local vision-driven desktop automation agent. Its principal workflow accepts a user goal, captures the current desktop, asks a locally loaded vision-language model for one JSON action, parses and dispatches that action, then repeats until completion or a stopping condition. A separate script supports one-off image queries. The graph reflects the main loop and its screenshot/input boundaries; model inference and parsing are distinct responsibilities within the same source file.
Main entry points: optimized_qwen_vision.py Agent Loop · snapshot.py Screenshot Capture · Qwen_vision.py Standalone Image Query · Mouse_use.py Mouse Keyboard Controls
Components 11
Agent Workflow
- Agent Loop
optimized_qwen_vision.py - JSON Action Parser
optimized_qwen_vision.py - Action Dispatcher
optimized_qwen_vision.py
Vision Reasoning
- Standalone Image Query
Qwen_vision.py - Vision Inference
optimized_qwen_vision.py - Local Qwen Model
Desktop I/O
- Screenshot Capture
snapshot.py - Mouse Keyboard Controls
Mouse_use.py
Other components
- User Goal
- Desktop Screen
- PyAutoGUI
Connections 12
- User Goal → Agent Loop: provides goal
README.md - Agent Loop → Screenshot Capture: captures screen
optimized_qwen_vision.py - Screenshot Capture → Desktop Screen: grabs image
snapshot.py - Agent Loop → Vision Inference: requests action
optimized_qwen_vision.py - Vision Inference → Local Qwen Model: generates response
optimized_qwen_vision.py - Vision Inference → JSON Action Parser: parses response
optimized_qwen_vision.py - Agent Loop → Action Dispatcher: executes action
optimized_qwen_vision.py - Action Dispatcher → Mouse Keyboard Controls: calls controls
optimized_qwen_vision.py - Action Dispatcher → PyAutoGUI: scrolls screen
optimized_qwen_vision.py - Mouse Keyboard Controls → PyAutoGUI: controls desktop
Mouse_use.py - Standalone Image Query → Local Qwen Model: runs vision query
Qwen_vision.py - Standalone Image Query → Screenshot Capture: captures screen
Qwen_vision.py