anionex/agent-vision-toolkit

Purpose and Entry Points

agent-vision-toolkit equips shell-capable, text-only coding agents with image understanding and visual workflows. Primary user entry points are executable CLIs in bin/ (bin/glance, bin/ground, bin/detect, bin/crop, bin/trace), the optional local HTTP bridge vision_proxy.py, and agent-native extensions under extensions/.

Shared Vision Client and Configuration

The Python vision boundary is centralized in vision_client.py, used by CLI and proxy paths to call an external OpenAI-compatible multimodal /chat/completions API with image URLs. Runtime configuration comes from VISION_API_KEY, VISION_BASE_URL, VISION_MODEL, and optional LANG (documented by .env.example). This separates the vision provider from the host text-model agent and preserves existing upstream text-model authentication.

CLI Vision Operations

The shell tools expose complementary visual primitives:

• Image Q&A/OCR: bin/glance delegates to the vision-client path.

• Target localization and screen inventory: bin/ground, bin/detect.

• Pixel-region extraction and deterministic vector recovery: bin/crop, bin/trace.

These form a composable loop: inspect an image, obtain original-image pixel coordinates, crop or zoom a region, then re-inspect, compare, or trace it. Core implementations are rooted at ground.py, detect.py, and bin/trace.

Long-Screenshot and Restoration Workflows

The repository packages higher-level agent procedures rather than only raw commands. skills/vision-tools/SKILL.md directs tool selection and sequencing; focused playbooks live in skills/vision-tools/references/long-screenshot-ocr.md, skills/vision-tools/references/restore-ui.md, and skills/vision-tools/references/gui.md. The long-image pipeline at skills/vision-tools/scripts/long_screenshot_ocr.py partitions screenshots into safe bands, OCRs chunks, merges overlap, and records boundary risk. These workflows let agents iteratively inspect and verify rather than trust a one-pass description.

Transparent Proxy Integration

vision_proxy.py is the optional seamless integration entry point for Codex and Claude Code. It accepts local agent traffic, forwards ordinary requests to the configured text-model upstream, and intercepts image-bearing requests. For intercepted images, it derives a focus hint from user intent or the assistant’s image-tool reason, calls the vision API, replaces image content with task-oriented text, then continues the original text-model request. The proxy is intentionally local and has process-lifetime caching; its default listener is described as 127.0.0.1:19100.

Native Agent Extensions

For runtimes that support native hooks rather than an HTTP proxy, single-file TypeScript integrations implement equivalent image rewriting:

• Pi / Oh My Pi: extensions/pi/vision.ts

• OpenCode: extensions/opencode/vision.ts

Their adjacent READMEs (extensions/pi/README.md, extensions/opencode/README.md) define host-specific installation and lifecycle boundaries. This keeps core vision configuration shared while adapting interception to each agent platform.

Runtime, Dependencies, and Supporting Surfaces

The main runtime is Python 3.11+ with optional image-processing dependencies: Pillow for image coordinate/crop flows, NumPy for tracing, and optional vtracer for outline fallback. Native extensions require the host’s TypeScript/plugin runtime. website/ is a separate Vite/React presentation site, rooted at website/src/main.jsx, and is not part of toolkit request processing. Tests in tests/ cover proxy rewriting, tool behavior, fail-open handling, and extension compatibility.