# damanegi2005/captcha_vlm_evaluation architecture

> Architecture diagram of the GitHub repository damanegi2005/captcha_vlm_evaluation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/damanegi2005/captcha_vlm_evaluation
- Repository: https://github.com/damanegi2005/captcha_vlm_evaluation

## Overview

This repository is a set of CAPTCHA VLM evaluation scripts, not a packaged service. The workflow reads CAPTCHA examples and ground-truth answers, prompts GPT-4.1-mini, compares predictions with answers using task-specific metrics, and saves evaluation results. The README also documents OCR, object-selection, and coordinate-localization capabilities; their likely script paths appear in the tree, but those files were not sampled. The shown scripts combine prompting, scoring, and output in one runtime module, so the graph preserves those boundaries rather than inventing shared pipeline layers.

Main entry points: [`test_ocr.py`](https://github.com/damanegi2005/captcha_vlm_evaluation/blob/main/test_ocr.py) (OCR evaluation), [`test_object.py`](https://github.com/damanegi2005/captcha_vlm_evaluation/blob/main/test_object.py) (Object selection), [`click_order.py`](https://github.com/damanegi2005/captcha_vlm_evaluation/blob/main/click_order.py) (Click order), [`arithmetic.py`](https://github.com/damanegi2005/captcha_vlm_evaluation/blob/main/arithmetic.py) (Arithmetic)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_tasks["CAPTCHA tasks"]
  node_ocr["OCR evaluation<br/>[test_ocr.py]"]
  node_object["Object selection<br/>[test_object.py]"]
  node_click["Click order<br/>[click_order.py]"]
  node_coordinate["Coordinate localization<br/>[click_order.py]"]
  node_arithmetic["Arithmetic<br/>[arithmetic.py]"]
  node_common["Common sense"]
  node_rules["Rule filtering<br/>[rule_based.py]"]
end

subgraph group_evaluation["Evaluation workflow"]
  node_dataset["CAPTCHA examples"]
  node_openai["GPT-4.1-mini API"]
  node_scoring["Task metrics"]
  node_results["Evaluation results"]
end

node_researcher(("Evaluator"))

node_researcher -->|"provides examples"| node_dataset
node_dataset -->|"supplies inputs"| node_click
node_dataset -->|"supplies inputs"| node_coordinate
node_dataset -->|"supplies inputs"| node_arithmetic
node_dataset -->|"supplies questions"| node_common
node_dataset -->|"supplies inputs"| node_rules
node_dataset -->|"supplies examples"| node_ocr
node_dataset -->|"supplies examples"| node_object
node_click -->|"requests predictions"| node_openai
node_coordinate -->|"requests predictions"| node_openai
node_arithmetic -->|"requests predictions"| node_openai
node_common -->|"requests predictions"| node_openai
node_rules -->|"requests predictions"| node_openai
node_ocr -->|"requests predictions"| node_openai
node_object -->|"requests predictions"| node_openai
node_click -->|"scores predictions"| node_scoring
node_coordinate -->|"scores predictions"| node_scoring
node_arithmetic -->|"scores predictions"| node_scoring
node_common -->|"scores predictions"| node_scoring
node_rules -->|"scores predictions"| node_scoring
node_ocr -->|"scores predictions"| node_scoring
node_object -->|"scores predictions"| node_scoring
node_click -->|"writes results"| node_results
node_coordinate -->|"writes results"| node_results
node_arithmetic -->|"writes results"| node_results
node_common -->|"writes results"| node_results
node_rules -->|"writes results"| node_results
node_ocr -->|"saves results"| node_results
node_object -->|"saves results"| node_results

click node_ocr "https://github.com/damanegi2005/captcha_vlm_evaluation/blob/main/test_ocr.py"
click node_object "https://github.com/damanegi2005/captcha_vlm_evaluation/blob/main/test_object.py"
click node_click "https://github.com/damanegi2005/captcha_vlm_evaluation/blob/main/click_order.py"
click node_coordinate "https://github.com/damanegi2005/captcha_vlm_evaluation/blob/main/click_order.py"
click node_arithmetic "https://github.com/damanegi2005/captcha_vlm_evaluation/blob/main/arithmetic.py"
click node_common "https://github.com/damanegi2005/captcha_vlm_evaluation/blob/main/Common_sense_reasoning.py"
click node_rules "https://github.com/damanegi2005/captcha_vlm_evaluation/blob/main/rule_based.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ocr,node_object,node_click,node_coordinate,node_arithmetic,node_common,node_rules toneBlue
class node_dataset,node_openai,node_scoring,node_results toneAmber
class node_researcher toneIndigo
```

## Components

### CAPTCHA tasks

- [OCR evaluation](https://github.com/damanegi2005/captcha_vlm_evaluation/blob/main/test_ocr.py): `test_ocr.py`
- [Object selection](https://github.com/damanegi2005/captcha_vlm_evaluation/blob/main/test_object.py): `test_object.py`
- [Click order](https://github.com/damanegi2005/captcha_vlm_evaluation/blob/main/click_order.py): `click_order.py`
- [Coordinate localization](https://github.com/damanegi2005/captcha_vlm_evaluation/blob/main/click_order.py): `click_order.py`
- [Arithmetic](https://github.com/damanegi2005/captcha_vlm_evaluation/blob/main/arithmetic.py): `arithmetic.py`
- [Common sense](https://github.com/damanegi2005/captcha_vlm_evaluation/blob/main/Common_sense_reasoning.py): `Common_sense_reasoning.py`
- [Rule filtering](https://github.com/damanegi2005/captcha_vlm_evaluation/blob/main/rule_based.py): `rule_based.py`

### Evaluation workflow

- **CAPTCHA examples**
- **GPT-4.1-mini API**
- **Task metrics**
- **Evaluation results**

### Other components

- **Evaluator**

## Connections

- Evaluator → CAPTCHA examples: provides examples (evidence: `README.md`)
- CAPTCHA examples → Click order: supplies inputs (evidence: `click_order.py`)
- CAPTCHA examples → Coordinate localization: supplies inputs (evidence: `click_order.py`)
- CAPTCHA examples → Arithmetic: supplies inputs (evidence: `arithmetic.py`)
- CAPTCHA examples → Common sense: supplies questions (evidence: `Common_sense_reasoning.py`)
- CAPTCHA examples → Rule filtering: supplies inputs (evidence: `rule_based.py`)
- CAPTCHA examples → OCR evaluation: supplies examples (evidence: `README.md`)
- CAPTCHA examples → Object selection: supplies examples (evidence: `README.md`)
- Click order → GPT-4.1-mini API: requests predictions (evidence: `click_order.py`)
- Coordinate localization → GPT-4.1-mini API: requests predictions (evidence: `click_order.py`)
- Arithmetic → GPT-4.1-mini API: requests predictions (evidence: `arithmetic.py`)
- Common sense → GPT-4.1-mini API: requests predictions (evidence: `Common_sense_reasoning.py`)
- Rule filtering → GPT-4.1-mini API: requests predictions (evidence: `rule_based.py`)
- OCR evaluation → GPT-4.1-mini API: requests predictions (evidence: `README.md`)
- Object selection → GPT-4.1-mini API: requests predictions (evidence: `README.md`)
- Click order → Task metrics: scores predictions (evidence: `click_order.py`)
- Coordinate localization → Task metrics: scores predictions (evidence: `click_order.py`)
- Arithmetic → Task metrics: scores predictions (evidence: `arithmetic.py`)
- Common sense → Task metrics: scores predictions (evidence: `Common_sense_reasoning.py`)
- Rule filtering → Task metrics: scores predictions (evidence: `rule_based.py`)
- OCR evaluation → Task metrics: scores predictions (evidence: `README.md`)
- Object selection → Task metrics: scores predictions (evidence: `README.md`)
- Click order → Evaluation results: writes results (evidence: `click_order.py`)
- Coordinate localization → Evaluation results: writes results (evidence: `click_order.py`)
- Arithmetic → Evaluation results: writes results (evidence: `arithmetic.py`)
- Common sense → Evaluation results: writes results (evidence: `Common_sense_reasoning.py`)
- Rule filtering → Evaluation results: writes results (evidence: `rule_based.py`)
- OCR evaluation → Evaluation results: saves results (evidence: `README.md`)
- Object selection → Evaluation results: saves results (evidence: `README.md`)