# vchitect/shotbench architecture

> Architecture diagram of the GitHub repository vchitect/shotbench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vchitect/shotbench
- Repository: https://github.com/vchitect/shotbench

## Overview

ShotBench’s checked-in product is an evaluation workflow: run ShotVL against the ShotBench test set to produce predictions, then calculate per-category and overall accuracy. The scorer first attempts to match answers directly and can optionally use OpenAI for ambiguous answers. The README describes broader benchmark, dataset, and model-training capabilities, but this tree contains no training or dataset-generation implementation. The map is limited to the two sampled runtime scripts and documented external inputs; it does not infer unsampled internals.

Main entry points: [`evaluation/shotvl/evaluate.py`](https://github.com/vchitect/shotbench/blob/main/evaluation/shotvl/evaluate.py) (ShotVL evaluation), [`evaluation/calculate_scores.py`](https://github.com/vchitect/shotbench/blob/main/evaluation/calculate_scores.py) (Score calculation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inference["Model inference"]
  node_shotvl_eval["ShotVL evaluation<br/>[evaluate.py]"]
end

subgraph group_scoring["Answer scoring"]
  node_predictions["Prediction workbook"]
  node_score_calc["Score calculation"]
  node_scores["Accuracy results"]
end

node_user(("Evaluator"))
node_testset[("ShotBench test set")]
node_shotvl_model["ShotVL model"]
node_openai["OpenAI API"]

node_user -->|"runs evaluation"| node_shotvl_eval
node_testset -->|"provides test data"| node_shotvl_eval
node_shotvl_eval -->|"generates answers"| node_shotvl_model
node_shotvl_eval -->|"writes predictions"| node_predictions
node_user -->|"runs scoring"| node_score_calc
node_predictions -->|"loads predictions"| node_score_calc
node_score_calc -->|"matches ambiguous answers"| node_openai
node_score_calc -->|"writes accuracy"| node_scores

click node_shotvl_eval "https://github.com/vchitect/shotbench/blob/main/evaluation/shotvl/evaluate.py"
click node_score_calc "https://github.com/vchitect/shotbench/blob/main/evaluation/calculate_scores.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_shotvl_eval toneBlue
class node_predictions,node_score_calc,node_scores,node_testset toneAmber
class node_user,node_shotvl_model toneIndigo
class node_openai toneMint
```

## Components

### Model inference

- [ShotVL evaluation](https://github.com/vchitect/shotbench/blob/main/evaluation/shotvl/evaluate.py): `evaluation/shotvl/evaluate.py`

### Answer scoring

- **Prediction workbook**
- [Score calculation](https://github.com/vchitect/shotbench/blob/main/evaluation/calculate_scores.py): `evaluation/calculate_scores.py`
- **Accuracy results**

### Other components

- **Evaluator**
- **ShotBench test set**
- **ShotVL model**
- **OpenAI API**

## Connections

- Evaluator → ShotVL evaluation: runs evaluation (evidence: `README.md`)
- ShotBench test set → ShotVL evaluation: provides test data (evidence: `evaluation/shotvl/evaluate.py`)
- ShotVL evaluation → ShotVL model: generates answers (evidence: `evaluation/shotvl/evaluate.py`)
- ShotVL evaluation → Prediction workbook: writes predictions (evidence: `evaluation/shotvl/evaluate.py`)
- Evaluator → Score calculation: runs scoring (evidence: `README.md`)
- Prediction workbook → Score calculation: loads predictions (evidence: `evaluation/calculate_scores.py`)
- Score calculation → OpenAI API: matches ambiguous answers (evidence: `evaluation/calculate_scores.py`)
- Score calculation → Accuracy results: writes accuracy (evidence: `evaluation/calculate_scores.py`)
