# anionex/agent-vision-toolkit architecture

> Architecture diagram of the GitHub repository anionex/agent-vision-toolkit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anionex/agent-vision-toolkit
- Repository: https://github.com/anionex/agent-vision-toolkit

## Overview

#### Purpose and Entry Points
`agent-vision-toolkit` equips shell-capable, text-only coding agents with image understanding and visual workflows. Primary user entry points are executable CLIs in `bin/` (`bin/glance`, `bin/ground`, `bin/detect`, `bin/crop`, `bin/trace`), the optional local HTTP bridge `vision_proxy.py`, and agent-native extensions under `extensions/`.

#### Shared Vision Client and Configuration
The Python vision boundary is centralized in `vision_client.py`, used by CLI and proxy paths to call an external OpenAI-compatible multimodal `/chat/completions` API with image URLs. Runtime configuration comes from `VISION_API_KEY`, `VISION_BASE_URL`, `VISION_MODEL`, and optional `LANG` (documented by `.env.example`). This separates the vision provider from the host text-model agent and preserves existing upstream text-model authentication.

#### CLI Vision Operations
The shell tools expose complementary visual primitives:
- Image Q&A/OCR: `bin/glance` delegates to the vision-client path.
- Target localization and screen inventory: `bin/ground`, `bin/detect`.
- Pixel-region extraction and deterministic vector recovery: `bin/crop`, `bin/trace`.

These form a composable loop: inspect an image, obtain original-image pixel coordinates, crop or zoom a region, then re-inspect, compare, or trace it. Core implementations are rooted at `ground.py`, `detect.py`, and `bin/trace`.

#### Long-Screenshot and Restoration Workflows
The repository packages higher-level agent procedures rather than only raw commands. `skills/vision-tools/SKILL.md` directs tool selection and sequencing; focused playbooks live in `skills/vision-tools/references/long-screenshot-ocr.md`, `skills/vision-tools/references/restore-ui.md`, and `skills/vision-tools/references/gui.md`. The long-image pipeline at `skills/vision-tools/scripts/long_screenshot_ocr.py` partitions screenshots into safe bands, OCRs chunks, merges overlap, and records boundary risk. These workflows let agents iteratively inspect and verify rather than trust a one-pass description.

#### Transparent Proxy Integration
`vision_proxy.py` is the optional seamless integration entry point for Codex and Claude Code. It accepts local agent traffic, forwards ordinary requests to the configured text-model upstream, and intercepts image-bearing requests. For intercepted images, it derives a focus hint from user intent or the assistant’s image-tool reason, calls the vision API, replaces image content with task-oriented text, then continues the original text-model request. The proxy is intentionally local and has process-lifetime caching; its default listener is described as `127.0.0.1:19100`.

#### Native Agent Extensions
For runtimes that support native hooks rather than an HTTP proxy, single-file TypeScript integrations implement equivalent image rewriting:
- Pi / Oh My Pi: `extensions/pi/vision.ts`
- OpenCode: `extensions/opencode/vision.ts`

Their adjacent READMEs (`extensions/pi/README.md`, `extensions/opencode/README.md`) define host-specific installation and lifecycle boundaries. This keeps core vision configuration shared while adapting interception to each agent platform.

#### Runtime, Dependencies, and Supporting Surfaces
The main runtime is Python 3.11+ with optional image-processing dependencies: Pillow for image coordinate/crop flows, NumPy for tracing, and optional `vtracer` for outline fallback. Native extensions require the host’s TypeScript/plugin runtime. `website/` is a separate Vite/React presentation site, rooted at `website/src/main.jsx`, and is not part of toolkit request processing. Tests in `tests/` cover proxy rewriting, tool behavior, fail-open handling, and extension compatibility.

Main entry points: [`vision_client.py`](https://github.com/Anionex/agent-vision-toolkit/blob/main/vision_client.py) (Vision client), [`skills/vision-tools/SKILL.md`](https://github.com/Anionex/agent-vision-toolkit/blob/main/skills/vision-tools/SKILL.md) (Vision tools skill), [`vision_proxy.py`](https://github.com/Anionex/agent-vision-toolkit/blob/main/vision_proxy.py) (Local vision proxy), [`bin/glance`](https://github.com/Anionex/agent-vision-toolkit/blob/main/bin/glance) (glance)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry Points"]
  node_glance["glance<br/>CLI"]
  node_ground["ground<br/>CLI"]
  node_detect["detect<br/>CLI"]
  node_crop["crop<br/>CLI"]
  node_trace["trace<br/>CLI"]
end

subgraph group_vision["Vision Core"]
  node_vision_client{{"Vision client<br/>Python API client<br/>[vision_client.py]"}}
  node_vision_config["Vision environment<br/>runtime config<br/>[.env.example]"]
  node_vision_api(("Vision provider<br/>external multimodal API"))
  node_ground_core["Grounding core<br/>Python operation<br/>[ground.py]"]
  node_detect_core["Detection core<br/>Python operation<br/>[detect.py]"]
end

subgraph group_workflows["Agent Workflows"]
  node_skill["Vision tools skill<br/>agent procedure<br/>[SKILL.md]"]
  node_long_ocr["Long screenshot OCR<br/>Python pipeline"]
  node_restoration_playbook["Restoration playbook<br/>workflow guide<br/>[restore-ui.md]"]
end

subgraph group_integrations["Agent Integrations"]
  node_proxy{{"Local vision proxy<br/>Python HTTP bridge<br/>[vision_proxy.py]"}}
  node_text_upstream(("Text-model upstream<br/>external text API"))
  node_pi_extension["Pi integration<br/>TypeScript hook<br/>[vision.ts]"]
  node_opencode_extension["OpenCode integration<br/>TypeScript hook<br/>[vision.ts]"]
end

node_website["Presentation site<br/>Vite/React site<br/>[main.jsx]"]

node_glance -->|"asks / OCRs"| node_vision_client
node_ground -->|"runs"| node_ground_core
node_detect -->|"runs"| node_detect_core
node_ground_core -->|"uses vision"| node_vision_client
node_detect_core -->|"uses vision"| node_vision_client
node_vision_config -.->|"configures"| node_vision_client
node_vision_client -->|"image requests"| node_vision_api
node_skill -.->|"guides use"| node_glance
node_skill -.->|"guides use"| node_ground
node_skill -.->|"guides use"| node_detect
node_skill -.->|"guides use"| node_crop
node_skill -.->|"guides use"| node_trace
node_skill -->|"invokes"| node_long_ocr
node_restoration_playbook -.->|"specializes"| node_skill
node_proxy -->|"intercepted images"| node_vision_client
node_proxy -->|"forwards rewritten requests"| node_text_upstream
node_pi_extension -->|"calls for images"| node_vision_client
node_opencode_extension -->|"calls for images"| node_vision_client

click node_glance "https://github.com/Anionex/agent-vision-toolkit/blob/main/bin/glance"
click node_ground "https://github.com/Anionex/agent-vision-toolkit/blob/main/bin/ground"
click node_detect "https://github.com/Anionex/agent-vision-toolkit/blob/main/bin/detect"
click node_crop "https://github.com/Anionex/agent-vision-toolkit/blob/main/bin/crop"
click node_trace "https://github.com/Anionex/agent-vision-toolkit/blob/main/bin/trace"
click node_vision_client "https://github.com/Anionex/agent-vision-toolkit/blob/main/vision_client.py"
click node_vision_config "https://github.com/Anionex/agent-vision-toolkit/blob/main/.env.example"
click node_ground_core "https://github.com/Anionex/agent-vision-toolkit/blob/main/ground.py"
click node_detect_core "https://github.com/Anionex/agent-vision-toolkit/blob/main/detect.py"
click node_skill "https://github.com/Anionex/agent-vision-toolkit/blob/main/skills/vision-tools/SKILL.md"
click node_long_ocr "https://github.com/Anionex/agent-vision-toolkit/blob/main/skills/vision-tools/scripts/long_screenshot_ocr.py"
click node_restoration_playbook "https://github.com/Anionex/agent-vision-toolkit/blob/main/skills/vision-tools/references/restore-ui.md"
click node_proxy "https://github.com/Anionex/agent-vision-toolkit/blob/main/vision_proxy.py"
click node_pi_extension "https://github.com/Anionex/agent-vision-toolkit/blob/main/extensions/pi/vision.ts"
click node_opencode_extension "https://github.com/Anionex/agent-vision-toolkit/blob/main/extensions/opencode/vision.ts"
click node_website "https://github.com/Anionex/agent-vision-toolkit/blob/main/website/src/main.jsx"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_glance,node_ground,node_detect,node_crop,node_trace toneBlue
class node_vision_client,node_vision_config,node_vision_api,node_ground_core,node_detect_core toneAmber
class node_skill,node_long_ocr,node_restoration_playbook toneMint
class node_proxy,node_text_upstream,node_pi_extension,node_opencode_extension toneRose
class node_website toneNeutral
```

## Components

### Entry Points

- [glance](https://github.com/Anionex/agent-vision-toolkit/blob/main/bin/glance): CLI · Image Q&A and OCR. · `bin/glance`
- [ground](https://github.com/Anionex/agent-vision-toolkit/blob/main/bin/ground): CLI · Pixel target localization. · `bin/ground`
- [detect](https://github.com/Anionex/agent-vision-toolkit/blob/main/bin/detect): CLI · Screen inventory and detection. · `bin/detect`
- [crop](https://github.com/Anionex/agent-vision-toolkit/blob/main/bin/crop): CLI · Pixel-region extraction. · `bin/crop`
- [trace](https://github.com/Anionex/agent-vision-toolkit/blob/main/bin/trace): CLI · Deterministic vector recovery. · `bin/trace`

### Vision Core

- [Vision client](https://github.com/Anionex/agent-vision-toolkit/blob/main/vision_client.py): Python API client · Shared OpenAI-compatible multimodal API boundary. · `vision_client.py`
- [Vision environment](https://github.com/Anionex/agent-vision-toolkit/blob/main/.env.example): runtime config · Provider key, base URL, model, and locale. · `.env.example`
- **Vision provider**: external multimodal API · OpenAI-compatible /chat/completions endpoint.
- [Grounding core](https://github.com/Anionex/agent-vision-toolkit/blob/main/ground.py): Python operation · `ground.py`
- [Detection core](https://github.com/Anionex/agent-vision-toolkit/blob/main/detect.py): Python operation · `detect.py`

### Agent Workflows

- [Vision tools skill](https://github.com/Anionex/agent-vision-toolkit/blob/main/skills/vision-tools/SKILL.md): agent procedure · Selects and sequences visual tools. · `skills/vision-tools/SKILL.md`
- [Long screenshot OCR](https://github.com/Anionex/agent-vision-toolkit/blob/main/skills/vision-tools/scripts/long_screenshot_ocr.py): Python pipeline · Bands, OCRs, merges overlap, and records boundary risk. · `skills/vision-tools/scripts/long_screenshot_ocr.py`
- [Restoration playbook](https://github.com/Anionex/agent-vision-toolkit/blob/main/skills/vision-tools/references/restore-ui.md): workflow guide · `skills/vision-tools/references/restore-ui.md`

### Agent Integrations

- [Local vision proxy](https://github.com/Anionex/agent-vision-toolkit/blob/main/vision_proxy.py): Python HTTP bridge · Rewrites image requests and caches for its process lifetime. · `vision_proxy.py`
- **Text-model upstream**: external text API · Receives ordinary and rewritten requests.
- [Pi integration](https://github.com/Anionex/agent-vision-toolkit/blob/main/extensions/pi/vision.ts): TypeScript hook · Pi and Oh My Pi image rewriting. · `extensions/pi/vision.ts`
- [OpenCode integration](https://github.com/Anionex/agent-vision-toolkit/blob/main/extensions/opencode/vision.ts): TypeScript hook · Host-native image rewriting. · `extensions/opencode/vision.ts`

### Other components

- [Presentation site](https://github.com/Anionex/agent-vision-toolkit/blob/main/website/src/main.jsx): Vite/React site · Separate from toolkit request processing. · `website/src/main.jsx`

## Connections

- glance → Vision client: asks / OCRs
- ground → Grounding core: runs
- detect → Detection core: runs
- Grounding core → Vision client: uses vision
- Detection core → Vision client: uses vision
- Vision environment → Vision client: configures
- Vision client → Vision provider: image requests
- Vision tools skill → glance: guides use
- Vision tools skill → ground: guides use
- Vision tools skill → detect: guides use
- Vision tools skill → crop: guides use
- Vision tools skill → trace: guides use
- Vision tools skill → Long screenshot OCR: invokes
- Restoration playbook → Vision tools skill: specializes
- Local vision proxy → Vision client: intercepted images
- Local vision proxy → Text-model upstream: forwards rewritten requests
- Pi integration → Vision client: calls for images
- OpenCode integration → Vision client: calls for images