How vchitect/shotbench works
ShotBench’s checked-in product is an evaluation workflow: run ShotVL against the ShotBench test set to produce predictions, then calculate per-category and overall accuracy. The scorer first attempts to match answers directly and can optionally use OpenAI for ambiguous answers. The README describes broader benchmark, dataset, and model-training capabilities, but this tree contains no training or dataset-generation implementation. The map is limited to the two sampled runtime scripts and documented external inputs; it does not infer unsampled internals.
Main entry points: evaluate.py ShotVL evaluation · calculate_scores.py Score calculation
Components 8
Model inference
- ShotVL evaluation
evaluation/shotvl/evaluate.py
Answer scoring
- Prediction workbook
- Score calculation
evaluation/calculate_scores.py - Accuracy results
Other components
- Evaluator
- ShotBench test set
- ShotVL model
- OpenAI API
Connections 8
- Evaluator → ShotVL evaluation: runs evaluation
README.md - ShotBench test set → ShotVL evaluation: provides test data
evaluation/shotvl/evaluate.py - ShotVL evaluation → ShotVL model: generates answers
evaluation/shotvl/evaluate.py - ShotVL evaluation → Prediction workbook: writes predictions
evaluation/shotvl/evaluate.py - Evaluator → Score calculation: runs scoring
README.md - Prediction workbook → Score calculation: loads predictions
evaluation/calculate_scores.py - Score calculation → OpenAI API: matches ambiguous answers
evaluation/calculate_scores.py - Score calculation → Accuracy results: writes accuracy
evaluation/calculate_scores.py