# faizan6xyz/agent-for-local-tasks_using_vision_model architecture

> Architecture diagram of the GitHub repository faizan6xyz/agent-for-local-tasks_using_vision_model, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/faizan6xyz/agent-for-local-tasks_using_vision_model
- Repository: https://github.com/faizan6xyz/agent-for-local-tasks_using_vision_model

## Overview

This repository is a local vision-driven desktop automation agent. Its principal workflow accepts a user goal, captures the current desktop, asks a locally loaded vision-language model for one JSON action, parses and dispatches that action, then repeats until completion or a stopping condition. A separate script supports one-off image queries. The graph reflects the main loop and its screenshot/input boundaries; model inference and parsing are distinct responsibilities within the same source file.

Main entry points: [`optimized_qwen_vision.py`](https://github.com/faizan6xyz/agent-for-local-tasks_using_vision_model/blob/main/optimized_qwen_vision.py) (Agent Loop), [`snapshot.py`](https://github.com/faizan6xyz/agent-for-local-tasks_using_vision_model/blob/main/snapshot.py) (Screenshot Capture), [`Qwen_vision.py`](https://github.com/faizan6xyz/agent-for-local-tasks_using_vision_model/blob/main/Qwen_vision.py) (Standalone Image Query), [`Mouse_use.py`](https://github.com/faizan6xyz/agent-for-local-tasks_using_vision_model/blob/main/Mouse_use.py) (Mouse Keyboard Controls)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_agent["Agent Workflow"]
  node_agent_loop["Agent Loop"]
  node_action_parser["JSON Action Parser"]
  node_action_dispatch["Action Dispatcher"]
end

subgraph group_vision["Vision Reasoning"]
  node_standalone_query["Standalone Image Query<br/>[Qwen_vision.py]"]
  node_vision_inference["Vision Inference"]
  node_local_model["Local Qwen Model"]
end

subgraph group_desktop["Desktop I/O"]
  node_screenshot_capture["Screenshot Capture<br/>[snapshot.py]"]
  node_input_controls["Mouse Keyboard Controls<br/>[Mouse_use.py]"]
end

node_user(("User Goal"))
node_desktop["Desktop Screen"]
node_pyautogui["PyAutoGUI"]

node_user -->|"provides goal"| node_agent_loop
node_agent_loop -->|"captures screen"| node_screenshot_capture
node_screenshot_capture -->|"grabs image"| node_desktop
node_agent_loop -->|"requests action"| node_vision_inference
node_vision_inference -->|"generates response"| node_local_model
node_vision_inference -->|"parses response"| node_action_parser
node_agent_loop -->|"executes action"| node_action_dispatch
node_action_dispatch -->|"calls controls"| node_input_controls
node_action_dispatch -->|"scrolls screen"| node_pyautogui
node_input_controls -->|"controls desktop"| node_pyautogui
node_standalone_query -->|"runs vision query"| node_local_model
node_standalone_query -->|"captures screen"| node_screenshot_capture

click node_agent_loop "https://github.com/faizan6xyz/agent-for-local-tasks_using_vision_model/blob/main/optimized_qwen_vision.py"
click node_action_parser "https://github.com/faizan6xyz/agent-for-local-tasks_using_vision_model/blob/main/optimized_qwen_vision.py"
click node_action_dispatch "https://github.com/faizan6xyz/agent-for-local-tasks_using_vision_model/blob/main/optimized_qwen_vision.py"
click node_standalone_query "https://github.com/faizan6xyz/agent-for-local-tasks_using_vision_model/blob/main/Qwen_vision.py"
click node_vision_inference "https://github.com/faizan6xyz/agent-for-local-tasks_using_vision_model/blob/main/optimized_qwen_vision.py"
click node_screenshot_capture "https://github.com/faizan6xyz/agent-for-local-tasks_using_vision_model/blob/main/snapshot.py"
click node_input_controls "https://github.com/faizan6xyz/agent-for-local-tasks_using_vision_model/blob/main/Mouse_use.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_agent_loop,node_action_parser,node_action_dispatch,node_user,node_desktop,node_pyautogui toneBlue
class node_standalone_query,node_vision_inference,node_local_model toneAmber
class node_screenshot_capture,node_input_controls toneMint
```

## Components

### Agent Workflow

- [Agent Loop](https://github.com/faizan6xyz/agent-for-local-tasks_using_vision_model/blob/main/optimized_qwen_vision.py): `optimized_qwen_vision.py`
- [JSON Action Parser](https://github.com/faizan6xyz/agent-for-local-tasks_using_vision_model/blob/main/optimized_qwen_vision.py): `optimized_qwen_vision.py`
- [Action Dispatcher](https://github.com/faizan6xyz/agent-for-local-tasks_using_vision_model/blob/main/optimized_qwen_vision.py): `optimized_qwen_vision.py`

### Vision Reasoning

- [Standalone Image Query](https://github.com/faizan6xyz/agent-for-local-tasks_using_vision_model/blob/main/Qwen_vision.py): `Qwen_vision.py`
- [Vision Inference](https://github.com/faizan6xyz/agent-for-local-tasks_using_vision_model/blob/main/optimized_qwen_vision.py): `optimized_qwen_vision.py`
- **Local Qwen Model**

### Desktop I/O

- [Screenshot Capture](https://github.com/faizan6xyz/agent-for-local-tasks_using_vision_model/blob/main/snapshot.py): `snapshot.py`
- [Mouse Keyboard Controls](https://github.com/faizan6xyz/agent-for-local-tasks_using_vision_model/blob/main/Mouse_use.py): `Mouse_use.py`

### Other components

- **User Goal**
- **Desktop Screen**
- **PyAutoGUI**

## Connections

- User Goal → Agent Loop: provides goal (evidence: `README.md`)
- Agent Loop → Screenshot Capture: captures screen (evidence: `optimized_qwen_vision.py`)
- Screenshot Capture → Desktop Screen: grabs image (evidence: `snapshot.py`)
- Agent Loop → Vision Inference: requests action (evidence: `optimized_qwen_vision.py`)
- Vision Inference → Local Qwen Model: generates response (evidence: `optimized_qwen_vision.py`)
- Vision Inference → JSON Action Parser: parses response (evidence: `optimized_qwen_vision.py`)
- Agent Loop → Action Dispatcher: executes action (evidence: `optimized_qwen_vision.py`)
- Action Dispatcher → Mouse Keyboard Controls: calls controls (evidence: `optimized_qwen_vision.py`)
- Action Dispatcher → PyAutoGUI: scrolls screen (evidence: `optimized_qwen_vision.py`)
- Mouse Keyboard Controls → PyAutoGUI: controls desktop (evidence: `Mouse_use.py`)
- Standalone Image Query → Local Qwen Model: runs vision query (evidence: `Qwen_vision.py`)
- Standalone Image Query → Screenshot Capture: captures screen (evidence: `Qwen_vision.py`)