# peiyu-georgia-li/crochetbench architecture

> Architecture diagram of the GitHub repository peiyu-georgia-li/crochetbench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/peiyu-georgia-li/crochetbench
- Repository: https://github.com/peiyu-georgia-li/crochetbench

## Overview

1. **Repository purpose and shape**
   - This is a benchmark repo, not an application runtime. Its core architecture is a pipeline for **collecting crochet patterns**, **building benchmark datasets**, **running multimodal model inference**, and **scoring outputs** across four tasks: recognition, selection, generation, and DSL synthesis.

2. **Primary data layer**
   - The `data/` directory is the fixed benchmark input layer.
   - It contains:
     - `crochet_pattern_by_project/` for task-facing pattern corpora organized by project category.
     - `mc_data.json.zip` for the multiple-choice dataset used in Task B.
     - `project_level_test.json` and `step_level_test_*.json` for Task D.
   - These files are the stable contract between dataset preparation and evaluation scripts.

3. **Data collection pipeline**
   - `collect_data/` is the upstream ingestion and dataset construction subsystem.
   - The flow is: scrape/collect patterns via `cli.py` and `data_collecting.py`, normalize them in `data_preprocessing.py`, attach images with `add_image.py`, then build the Task B multiple-choice dataset with `create_mc_dataset.py`.
   - This is the only part of the repo that transforms external crochet sources into benchmark-ready artifacts.

4. **Data analysis subsystem**
   - `data_analysis/` is separate from benchmark execution and is used for dataset inspection and quality checks.
   - It includes image-link validation, pattern-complexity analysis, and skill-level distribution analysis.
   - `unique_stitches.txt` acts as a reference vocabulary for stitch classification and helps define the label space for Task A.

5. **Benchmark execution layer**
   - `benchmark_task/` is the main runtime layer for all evaluations.
   - It contains per-task, per-model scripts such as `task_a_blip.py`, `task_c_gpt4o.py`, `task_d_project_qwen.py`, etc.
   - The architecture is intentionally replicated across models: each task has one script variant per model family, but the task logic is the stable boundary.

6. **Task decomposition**
   - **Task A**: image-to-stitch recognition; model outputs stitch labels from a crochet image.
   - **Task B**: image + multiple-choice instruction selection; the script includes its own scoring.
   - **Task C**: image-to-natural-language instruction generation; outputs text for later evaluation.
   - **Task D**: image + instructions-to-DSL translation, split into:
     - **step-level** incremental synthesis using prefix context
     - **project-level** full-program synthesis from complete instructions

7. **Evaluation boundary**
   - Evaluation is split between standalone scoring scripts and task scripts:
     - `eval_task_a.py`, `eval_task_c.py`, and the Task D eval scripts compute metrics from saved predictions.
     - Task B evaluation is embedded in the task script itself.
   - This creates a clean separation between **generation/inference** and **metric computation** for most tasks.

8. **DSL validation subsystem**
   - Task D depends on executable correctness, not just text similarity.
   - `verify_crochet_pattern.js` and `verify_crochet_pattern_with_history.js` provide validation of generated CrochetPARADE DSL.
   - These scripts sit at a language/runtime boundary: Python produces candidate DSL, and Node.js validates compilation/executability against the CrochetPARADE toolchain.

9. **External dependency boundary**
   - The README indicates a required clone of the **CrochetPARADE** repository for validation.
   - That means Task D is architecturally coupled to an external DSL compiler/validator, while the rest of the benchmark remains self-contained Python.
   - Closed-source model scripts also depend on API keys from `.env` for Anthropic, OpenAI, and Gemini.

10. **Model integration pattern**
   - The model scripts are adapters around a shared benchmark format.
   - Open-source models such as BLIP, Gemma, Qwen, and DeepSeek-VL are invoked locally via Python.
   - Closed-source models such as Claude, GPT-4o, and Gemini are invoked through API calls.
   - This means the repo has two inference backends, but both feed the same task-level output files.

11. **Task D data partitioning**
   - Step-level Task D uses three separate test slices: early, mid, and late steps.
   - This reflects an important architectural constraint: the benchmark treats incremental context as part of the problem, so the dataset is partitioned by pattern position rather than only by project.
   - Project-level Task D uses a different evaluation contract: full instruction set in, whole DSL program out.

12. **Fine-tuning subsystem**
   - `finetune/` is a secondary training path, not part of the main benchmark execution loop.
   - `train_qwen_task_c.py` and its requirements indicate an experiment-specific training pipeline, likely for adapting Qwen to Task C generation.
   - Architecturally, this sits beside benchmarking as an optional model adaptation stage before evaluation.

13. **Configuration and reproducibility**
   - `.env.example` defines the external service configuration boundary.
   - `requirements.txt` and `finetune/requirements.txt` show that the repo is primarily Python-based, with Node.js added only for DSL validation.
   - The benchmark assumes prebuilt data artifacts are present, so reproducibility depends on the shipped JSON files and any downloaded images.

14. **Diagram-friendly high-level flow**
   - External crochet sources → `collect_data/` ingestion/preprocessing → curated artifacts in `data/`
   - `data/` + images → task-specific model scripts in `benchmark_task/`
   - Model outputs → metric scripts or embedded evaluators
   - Task D outputs → CrochetPARADE/Node.js validator → compilation-based success metrics
   - Optional: `finetune/` can feed improved model checkpoints back into the same benchmark loop

Main entry points: [`data/crochet_pattern_by_project`](https://github.com/peiyu-georgia-li/crochetbench/tree/main/data/crochet_pattern_by_project) (Project corpus), [`collect_data/data_preprocessing.py`](https://github.com/peiyu-georgia-li/crochetbench/blob/main/collect_data/data_preprocessing.py) (Preprocess), [`benchmark_task/task_d_step_blip.py`](https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/task_d_step_blip.py) (Task D step), [`benchmark_task/task_d_project_blip.py`](https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/task_d_project_blip.py) (Task D project)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_data["Benchmark data"]
  node_data_patterns[("Project corpus<br/>pattern corpus")]
  node_data_mc[("MC dataset<br/>task B data<br/>[mc_data.json.zip]")]
  node_data_project_test["Project test<br/>task D project data"]
  node_data_step_test["Step test slices<br/>task D step data"]
end

subgraph group_group_collect["Data collection"]
  node_collect_cli["CLI ingest<br/>collector entrypoint<br/>[cli.py]"]
  node_collect_core["Collect data<br/>ingestion core<br/>[data_collecting.py]"]
  node_collect_preprocess["Preprocess<br/>normalization"]
  node_collect_images["Attach images<br/>enrichment<br/>[add_image.py]"]
  node_collect_mc["Build MC set<br/>dataset builder"]
end

subgraph group_group_benchmark["Benchmark execution"]
  node_bench_task_a["Task A<br/>recognition benchmark<br/>[task_a_blip.py]"]
  node_bench_task_b["Task B<br/>selection benchmark<br/>[task_b_blip.py]"]
  node_bench_task_c["Task C<br/>generation benchmark<br/>[task_c_blip.py]"]
  node_bench_task_d_step["Task D step<br/>step DSL benchmark"]
  node_bench_task_d_project["Task D project<br/>project DSL benchmark"]
  node_eval_a["Eval A<br/>metric script<br/>[eval_task_a.py]"]
  node_eval_c["Eval C<br/>metric script<br/>[eval_task_c.py]"]
  node_eval_d_step["Eval D step<br/>metric script"]
  node_eval_d_project["Eval D project<br/>metric script"]
end

subgraph group_group_analysis["Data analysis"]
  node_stitch_vocab["Stitch vocab<br/>label space"]
  node_analysis_links["Link check<br/>data QA"]
  node_analysis_complexity["Complexity stats<br/>data QA"]
  node_analysis_skill["Skill stats<br/>data QA"]
end

subgraph group_group_finetune["Fine-tuning"]
  node_finetune_qwen["Qwen train<br/>fine-tuning"]
end

subgraph group_group_external["External dependencies"]
  node_dsl_verify["DSL verify<br/>validator"]
  node_dsl_history["History verify<br/>validator"]
  node_crochetparade[("CrochetPARADE<br/>external toolchain")]
end

node_collect_cli -->|"ingests"| node_collect_core
node_collect_core -->|"normalizes"| node_collect_preprocess
node_collect_preprocess -->|"enriches"| node_collect_images
node_collect_images -->|"builds"| node_collect_mc
node_collect_preprocess -->|"writes"| node_data_patterns
node_collect_mc -->|"produces"| node_data_mc
node_collect_preprocess -->|"derives"| node_data_project_test
node_collect_preprocess -->|"derives"| node_data_step_test
node_data_patterns -->|"analyzed by"| node_analysis_complexity
node_data_patterns -->|"analyzed by"| node_analysis_skill
node_data_patterns -->|"checked by"| node_analysis_links
node_data_patterns -->|"references"| node_stitch_vocab
node_data_patterns -->|"input"| node_bench_task_a
node_data_mc -->|"input"| node_bench_task_b
node_data_patterns -->|"input"| node_bench_task_c
node_data_step_test -->|"input"| node_bench_task_d_step
node_data_project_test -->|"input"| node_bench_task_d_project
node_bench_task_a -->|"scores"| node_eval_a
node_bench_task_c -->|"scores"| node_eval_c
node_bench_task_d_step -->|"scores"| node_eval_d_step
node_bench_task_d_project -->|"scores"| node_eval_d_project
node_bench_task_b -.->|"embedded eval"| node_data_mc
node_bench_task_d_step -->|"validates"| node_dsl_history
node_bench_task_d_project -->|"validates"| node_dsl_verify
node_dsl_history -->|"depends on"| node_crochetparade
node_dsl_verify -->|"depends on"| node_crochetparade
node_finetune_qwen -.->|"feeds"| node_bench_task_c

click node_collect_cli "https://github.com/peiyu-georgia-li/crochetbench/blob/main/collect_data/cli.py"
click node_collect_core "https://github.com/peiyu-georgia-li/crochetbench/blob/main/collect_data/data_collecting.py"
click node_collect_preprocess "https://github.com/peiyu-georgia-li/crochetbench/blob/main/collect_data/data_preprocessing.py"
click node_collect_images "https://github.com/peiyu-georgia-li/crochetbench/blob/main/collect_data/add_image.py"
click node_collect_mc "https://github.com/peiyu-georgia-li/crochetbench/blob/main/collect_data/create_mc_dataset.py"
click node_data_patterns "https://github.com/peiyu-georgia-li/crochetbench/tree/main/data/crochet_pattern_by_project"
click node_data_mc "https://github.com/peiyu-georgia-li/crochetbench/blob/main/data/mc_data.json.zip"
click node_data_project_test "https://github.com/peiyu-georgia-li/crochetbench/blob/main/data/project_level_test.json"
click node_data_step_test "https://github.com/peiyu-georgia-li/crochetbench/blob/main/data/step_level_test_1_2.json"
click node_stitch_vocab "https://github.com/peiyu-georgia-li/crochetbench/blob/main/data_analysis/unique_stitches.txt"
click node_analysis_links "https://github.com/peiyu-georgia-li/crochetbench/blob/main/data_analysis/analyze_img_links.py"
click node_analysis_complexity "https://github.com/peiyu-georgia-li/crochetbench/blob/main/data_analysis/analyze_pattern_complexity.py"
click node_analysis_skill "https://github.com/peiyu-georgia-li/crochetbench/blob/main/data_analysis/analyze_skill_level.py"
click node_bench_task_a "https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/task_a_blip.py"
click node_bench_task_b "https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/task_b_blip.py"
click node_bench_task_c "https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/task_c_blip.py"
click node_bench_task_d_step "https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/task_d_step_blip.py"
click node_bench_task_d_project "https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/task_d_project_blip.py"
click node_eval_a "https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/eval_task_a.py"
click node_eval_c "https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/eval_task_c.py"
click node_eval_d_step "https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/eval_task_d_step_level.py"
click node_eval_d_project "https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/eval_task_d_project_level.py"
click node_dsl_verify "https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/verify_crochet_pattern.js"
click node_dsl_history "https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/verify_crochet_pattern_with_history.js"
click node_finetune_qwen "https://github.com/peiyu-georgia-li/crochetbench/blob/main/finetune/train_qwen_task_c.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_data_patterns,node_data_mc,node_data_project_test,node_data_step_test toneBlue
class node_collect_cli,node_collect_core,node_collect_preprocess,node_collect_images,node_collect_mc toneAmber
class node_bench_task_a,node_bench_task_b,node_bench_task_c,node_bench_task_d_step,node_bench_task_d_project,node_eval_a,node_eval_c,node_eval_d_step,node_eval_d_project toneMint
class node_stitch_vocab,node_analysis_links,node_analysis_complexity,node_analysis_skill toneRose
class node_finetune_qwen toneIndigo
class node_dsl_verify,node_dsl_history,node_crochetparade toneTeal
```

## Components

### Benchmark data

Stable benchmark inputs and curated task datasets.

- [Project corpus](https://github.com/peiyu-georgia-li/crochetbench/tree/main/data/crochet_pattern_by_project): pattern corpus · Per-project crochet pattern corpora used across tasks. · `data/crochet_pattern_by_project`
- [MC dataset](https://github.com/peiyu-georgia-li/crochetbench/blob/main/data/mc_data.json.zip): task B data · Packaged multiple-choice benchmark data for selection. · `data/mc_data.json.zip`
- [Project test](https://github.com/peiyu-georgia-li/crochetbench/blob/main/data/project_level_test.json): task D project data · Full-project test set for DSL synthesis. · `data/project_level_test.json`
- [Step test slices](https://github.com/peiyu-georgia-li/crochetbench/blob/main/data/step_level_test_1_2.json): task D step data · Incremental step-level test slices for prefix-conditioned DSL synthesis. · `data/step_level_test_1_2.json`

### Data collection

Upstream ingestion and dataset construction pipeline.

- [CLI ingest](https://github.com/peiyu-georgia-li/crochetbench/blob/main/collect_data/cli.py): collector entrypoint · Command-line entrypoint for crawling and collecting crochet patterns. · `collect_data/cli.py`
- [Collect data](https://github.com/peiyu-georgia-li/crochetbench/blob/main/collect_data/data_collecting.py): ingestion core · Fetches and structures source patterns from external crochet content. · `collect_data/data_collecting.py`
- [Preprocess](https://github.com/peiyu-georgia-li/crochetbench/blob/main/collect_data/data_preprocessing.py): normalization · Cleans and normalizes collected patterns into benchmark-ready form. · `collect_data/data_preprocessing.py`
- [Attach images](https://github.com/peiyu-georgia-li/crochetbench/blob/main/collect_data/add_image.py): enrichment · Adds image references to patterns for multimodal benchmark tasks. · `collect_data/add_image.py`
- [Build MC set](https://github.com/peiyu-georgia-li/crochetbench/blob/main/collect_data/create_mc_dataset.py): dataset builder · Constructs the Task B multiple-choice dataset. · `collect_data/create_mc_dataset.py`

### Benchmark execution

Per-task inference scripts and evaluation entrypoints.

- [Task A](https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/task_a_blip.py): recognition benchmark · Image-to-stitch recognition across model adapters. · `benchmark_task/task_a_blip.py`
- [Task B](https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/task_b_blip.py): selection benchmark · Image plus choices instruction selection with embedded scoring. · `benchmark_task/task_b_blip.py`
- [Task C](https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/task_c_blip.py): generation benchmark · Image-to-natural-language instruction generation. · `benchmark_task/task_c_blip.py`
- [Task D step](https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/task_d_step_blip.py): step DSL benchmark · Incremental DSL synthesis from partial context and image. · `benchmark_task/task_d_step_blip.py`
- [Task D project](https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/task_d_project_blip.py): project DSL benchmark · Full-program DSL synthesis from complete instructions and image. · `benchmark_task/task_d_project_blip.py`
- [Eval A](https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/eval_task_a.py): metric script · Computes recognition metrics from saved Task A predictions. · `benchmark_task/eval_task_a.py`
- [Eval C](https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/eval_task_c.py): metric script · Computes generation metrics from saved Task C outputs. · `benchmark_task/eval_task_c.py`
- [Eval D step](https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/eval_task_d_step_level.py): metric script · Evaluates step-level Task D outputs against references and executable checks. · `benchmark_task/eval_task_d_step_level.py`
- [Eval D project](https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/eval_task_d_project_level.py): metric script · Evaluates project-level Task D outputs against references and executable checks. · `benchmark_task/eval_task_d_project_level.py`

### Data analysis

Dataset inspection, validation, and quality analysis.

- [Stitch vocab](https://github.com/peiyu-georgia-li/crochetbench/blob/main/data_analysis/unique_stitches.txt): label space · Reference stitch vocabulary used for recognition and labeling. · `data_analysis/unique_stitches.txt`
- [Link check](https://github.com/peiyu-georgia-li/crochetbench/blob/main/data_analysis/analyze_img_links.py): data QA · Validates image-link integrity in the collected dataset. · `data_analysis/analyze_img_links.py`
- [Complexity stats](https://github.com/peiyu-georgia-li/crochetbench/blob/main/data_analysis/analyze_pattern_complexity.py): data QA · Analyzes pattern complexity and structural characteristics. · `data_analysis/analyze_pattern_complexity.py`
- [Skill stats](https://github.com/peiyu-georgia-li/crochetbench/blob/main/data_analysis/analyze_skill_level.py): data QA · Analyzes skill-level distribution across the benchmark corpus. · `data_analysis/analyze_skill_level.py`

### Fine-tuning

Optional model adaptation path outside the main benchmark loop.

- [Qwen train](https://github.com/peiyu-georgia-li/crochetbench/blob/main/finetune/train_qwen_task_c.py): fine-tuning · Optional training path for adapting Qwen to Task C generation. · `finetune/train_qwen_task_c.py`

### External dependencies

Non-repo runtimes and toolchains required by parts of the benchmark.

- [DSL verify](https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/verify_crochet_pattern.js): validator · Validates generated CrochetPARADE DSL for correctness and executability. · `benchmark_task/verify_crochet_pattern.js`
- [History verify](https://github.com/peiyu-georgia-li/crochetbench/blob/main/benchmark_task/verify_crochet_pattern_with_history.js): validator · Validates DSL with prior-step history for incremental synthesis. · `benchmark_task/verify_crochet_pattern_with_history.js`
- **CrochetPARADE**: external toolchain · External DSL compiler and validation runtime required by Task D.

## Connections

- CLI ingest → Collect data: ingests · CLI drives the collection workflow.
- Collect data → Preprocess: normalizes · Collected patterns are cleaned and structured.
- Preprocess → Attach images: enriches · Preprocessed records are paired with images.
- Attach images → Build MC set: builds · Enriched data feeds the multiple-choice dataset builder.
- Preprocess → Project corpus: writes · Prepared pattern corpora are stored as benchmark inputs.
- Build MC set → MC dataset: produces · Task B artifact is written to the benchmark data layer.
- Preprocess → Project test: derives · Project-level evaluation data comes from curated pattern sources.
- Preprocess → Step test slices: derives · Step-level evaluation slices are created from curated patterns.
- Project corpus → Complexity stats: analyzed by · Corpus supports complexity analysis.
- Project corpus → Skill stats: analyzed by · Corpus supports skill-level distribution analysis.
- Project corpus → Link check: checked by · Corpus images and links are validated.
- Project corpus → Stitch vocab: references · Vocabulary defines stitch labels for recognition.
- Project corpus → Task A: input · Task A consumes pattern images and labels.
- MC dataset → Task B: input · Task B consumes the packaged multiple-choice data.
- Project corpus → Task C: input · Task C consumes images for instruction generation.
- Step test slices → Task D step: input · Step-level Task D uses incremental test slices.
- Project test → Task D project: input · Project-level Task D uses full-program test data.
- Task A → Eval A: scores · Saved predictions are evaluated after inference.
- Task C → Eval C: scores · Generated instructions are scored offline.
- Task D step → Eval D step: scores · Step-level outputs flow into metric computation and validation.
- Task D project → Eval D project: scores · Project-level outputs flow into metric computation and validation.
- Task B → MC dataset: embedded eval · Task B performs scoring within the task script.
- Task D step → History verify: validates · Incremental DSL outputs are checked with history.
- Task D project → DSL verify: validates · Full DSL outputs are checked against the verifier.
- History verify → CrochetPARADE: depends on · History-aware validation requires the external CrochetPARADE toolchain.
- DSL verify → CrochetPARADE: depends on · DSL validation requires the external CrochetPARADE toolchain.
- Qwen train → Task C: feeds · Fine-tuned checkpoints can be reused in the Task C benchmark loop.
