How peiyu-georgia-li/crochetbench works
1. Repository purpose and shape
- This is a benchmark repo, not an application runtime. Its core architecture is a pipeline for collecting crochet patterns, building benchmark datasets, running multimodal model inference, and scoring outputs across four tasks: recognition, selection, generation, and DSL synthesis.
2. Primary data layer
- The data/ directory is the fixed benchmark input layer.
- It contains:
- crochet_pattern_by_project/ for task-facing pattern corpora organized by project category.
- mc_data.json.zip for the multiple-choice dataset used in Task B.
- project_level_test.json and step_level_test_*.json for Task D.
- These files are the stable contract between dataset preparation and evaluation scripts.
3. Data collection pipeline
- collect_data/ is the upstream ingestion and dataset construction subsystem.
- The flow is: scrape/collect patterns via cli.py and data_collecting.py, normalize them in data_preprocessing.py, attach images with add_image.py, then build the Task B multiple-choice dataset with create_mc_dataset.py.
- This is the only part of the repo that transforms external crochet sources into benchmark-ready artifacts.
4. Data analysis subsystem
- data_analysis/ is separate from benchmark execution and is used for dataset inspection and quality checks.
- It includes image-link validation, pattern-complexity analysis, and skill-level distribution analysis.
- unique_stitches.txt acts as a reference vocabulary for stitch classification and helps define the label space for Task A.
5. Benchmark execution layer
- benchmark_task/ is the main runtime layer for all evaluations.
- It contains per-task, per-model scripts such as task_a_blip.py, task_c_gpt4o.py, task_d_project_qwen.py, etc.
- The architecture is intentionally replicated across models: each task has one script variant per model family, but the task logic is the stable boundary.
6. Task decomposition
- Task A: image-to-stitch recognition; model outputs stitch labels from a crochet image.
- Task B: image + multiple-choice instruction selection; the script includes its own scoring.
- Task C: image-to-natural-language instruction generation; outputs text for later evaluation.
- Task D: image + instructions-to-DSL translation, split into:
- step-level incremental synthesis using prefix context
- project-level full-program synthesis from complete instructions
7. Evaluation boundary
- Evaluation is split between standalone scoring scripts and task scripts:
- eval_task_a.py, eval_task_c.py, and the Task D eval scripts compute metrics from saved predictions.
- Task B evaluation is embedded in the task script itself.
- This creates a clean separation between generation/inference and metric computation for most tasks.
8. DSL validation subsystem
- Task D depends on executable correctness, not just text similarity.
- verify_crochet_pattern.js and verify_crochet_pattern_with_history.js provide validation of generated CrochetPARADE DSL.
- These scripts sit at a language/runtime boundary: Python produces candidate DSL, and Node.js validates compilation/executability against the CrochetPARADE toolchain.
9. External dependency boundary
- The README indicates a required clone of the CrochetPARADE repository for validation.
- That means Task D is architecturally coupled to an external DSL compiler/validator, while the rest of the benchmark remains self-contained Python.
- Closed-source model scripts also depend on API keys from .env for Anthropic, OpenAI, and Gemini.
10. Model integration pattern
- The model scripts are adapters around a shared benchmark format.
- Open-source models such as BLIP, Gemma, Qwen, and DeepSeek-VL are invoked locally via Python.
- Closed-source models such as Claude, GPT-4o, and Gemini are invoked through API calls.
- This means the repo has two inference backends, but both feed the same task-level output files.
11. Task D data partitioning
- Step-level Task D uses three separate test slices: early, mid, and late steps.
- This reflects an important architectural constraint: the benchmark treats incremental context as part of the problem, so the dataset is partitioned by pattern position rather than only by project.
- Project-level Task D uses a different evaluation contract: full instruction set in, whole DSL program out.
12. Fine-tuning subsystem
- finetune/ is a secondary training path, not part of the main benchmark execution loop.
- train_qwen_task_c.py and its requirements indicate an experiment-specific training pipeline, likely for adapting Qwen to Task C generation.
- Architecturally, this sits beside benchmarking as an optional model adaptation stage before evaluation.
13. Configuration and reproducibility
- .env.example defines the external service configuration boundary.
- requirements.txt and finetune/requirements.txt show that the repo is primarily Python-based, with Node.js added only for DSL validation.
- The benchmark assumes prebuilt data artifacts are present, so reproducibility depends on the shipped JSON files and any downloaded images.
14. Diagram-friendly high-level flow
- External crochet sources → collect_data/ ingestion/preprocessing → curated artifacts in data/
- data/ + images → task-specific model scripts in benchmark_task/
- Model outputs → metric scripts or embedded evaluators
- Task D outputs → CrochetPARADE/Node.js validator → compilation-based success metrics
- Optional: finetune/ can feed improved model checkpoints back into the same benchmark loop
Main entry points: crochet_pattern_by_project Project corpus · data_preprocessing.py Preprocess · task_d_step_blip.py Task D step · task_d_project_blip.py Task D project
Components 26
Benchmark data
Stable benchmark inputs and curated task datasets.
- Project corpus · Per-project crochet pattern corpora used across tasks.
data/crochet_pattern_by_project - MC dataset · Packaged multiple-choice benchmark data for selection.
data/mc_data.json.zip - Project test · Full-project test set for DSL synthesis.
data/project_level_test.json - Step test slices · Incremental step-level test slices for prefix-conditioned DSL synthesis.
data/step_level_test_1_2.json
Data collection
Upstream ingestion and dataset construction pipeline.
- CLI ingest · Command-line entrypoint for crawling and collecting crochet patterns.
collect_data/cli.py - Collect data · Fetches and structures source patterns from external crochet content.
collect_data/data_collecting.py - Preprocess · Cleans and normalizes collected patterns into benchmark-ready form.
collect_data/data_preprocessing.py - Attach images · Adds image references to patterns for multimodal benchmark tasks.
collect_data/add_image.py - Build MC set · Constructs the Task B multiple-choice dataset.
collect_data/create_mc_dataset.py
Benchmark execution
Per-task inference scripts and evaluation entrypoints.
- Task A · Image-to-stitch recognition across model adapters.
benchmark_task/task_a_blip.py - Task B · Image plus choices instruction selection with embedded scoring.
benchmark_task/task_b_blip.py - Task C · Image-to-natural-language instruction generation.
benchmark_task/task_c_blip.py - Task D step · Incremental DSL synthesis from partial context and image.
benchmark_task/task_d_step_blip.py - Task D project · Full-program DSL synthesis from complete instructions and image.
benchmark_task/task_d_project_blip.py - Eval A · Computes recognition metrics from saved Task A predictions.
benchmark_task/eval_task_a.py - Eval C · Computes generation metrics from saved Task C outputs.
benchmark_task/eval_task_c.py - Eval D step · Evaluates step-level Task D outputs against references and executable checks.
benchmark_task/eval_task_d_step_level.py - Eval D project · Evaluates project-level Task D outputs against references and executable checks.
benchmark_task/eval_task_d_project_level.py
Data analysis
Dataset inspection, validation, and quality analysis.
- Stitch vocab · Reference stitch vocabulary used for recognition and labeling.
data_analysis/unique_stitches.txt - Link check · Validates image-link integrity in the collected dataset.
data_analysis/analyze_img_links.py - Complexity stats · Analyzes pattern complexity and structural characteristics.
data_analysis/analyze_pattern_complexity.py - Skill stats · Analyzes skill-level distribution across the benchmark corpus.
data_analysis/analyze_skill_level.py
Fine-tuning
Optional model adaptation path outside the main benchmark loop.
- Qwen train · Optional training path for adapting Qwen to Task C generation.
finetune/train_qwen_task_c.py
External dependencies
Non-repo runtimes and toolchains required by parts of the benchmark.
- DSL verify · Validates generated CrochetPARADE DSL for correctness and executability.
benchmark_task/verify_crochet_pattern.js - History verify · Validates DSL with prior-step history for incremental synthesis.
benchmark_task/verify_crochet_pattern_with_history.js - CrochetPARADE · External DSL compiler and validation runtime required by Task D.
Connections 27
- CLI ingest → Collect data: ingests · CLI drives the collection workflow.
- Collect data → Preprocess: normalizes · Collected patterns are cleaned and structured.
- Preprocess → Attach images: enriches · Preprocessed records are paired with images.
- Attach images → Build MC set: builds · Enriched data feeds the multiple-choice dataset builder.
- Preprocess → Project corpus: writes · Prepared pattern corpora are stored as benchmark inputs.
- Build MC set → MC dataset: produces · Task B artifact is written to the benchmark data layer.
- Preprocess → Project test: derives · Project-level evaluation data comes from curated pattern sources.
- Preprocess → Step test slices: derives · Step-level evaluation slices are created from curated patterns.
- Project corpus → Complexity stats: analyzed by · Corpus supports complexity analysis.
- Project corpus → Skill stats: analyzed by · Corpus supports skill-level distribution analysis.
- Project corpus → Link check: checked by · Corpus images and links are validated.
- Project corpus → Stitch vocab: references · Vocabulary defines stitch labels for recognition.
- Project corpus → Task A: input · Task A consumes pattern images and labels.
- MC dataset → Task B: input · Task B consumes the packaged multiple-choice data.
- Project corpus → Task C: input · Task C consumes images for instruction generation.
- Step test slices → Task D step: input · Step-level Task D uses incremental test slices.
- Project test → Task D project: input · Project-level Task D uses full-program test data.
- Task A → Eval A: scores · Saved predictions are evaluated after inference.
- Task C → Eval C: scores · Generated instructions are scored offline.
- Task D step → Eval D step: scores · Step-level outputs flow into metric computation and validation.
- Task D project → Eval D project: scores · Project-level outputs flow into metric computation and validation.
- Task B → MC dataset: embedded eval · Task B performs scoring within the task script.
- Task D step → History verify: validates · Incremental DSL outputs are checked with history.
- Task D project → DSL verify: validates · Full DSL outputs are checked against the verifier.
- History verify → CrochetPARADE: depends on · History-aware validation requires the external CrochetPARADE toolchain.
- DSL verify → CrochetPARADE: depends on · DSL validation requires the external CrochetPARADE toolchain.
- Qwen train → Task C: feeds · Fine-tuned checkpoints can be reused in the Task C benchmark loop.