# to2002td-cpu/dataset_builder_mel architecture

> Architecture diagram of the GitHub repository to2002td-cpu/dataset_builder_mel, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/to2002td-cpu/dataset_builder_mel
- Repository: https://github.com/to2002td-cpu/dataset_builder_mel

## Overview

1. **Repo purpose and top-level shape**  
   This repository is a data-building pipeline for a multimodal named entity disambiguation benchmark. It is organized as a multi-stage ETL system plus downstream analysis/evaluation tools. The core package is `wikiambig/`; the other top-level directories are consumers of its outputs: `split_gen/` for filtering a final split, `fig_gen/` for statistics/figures, and `eval/` for model evaluation.

2. **Primary architectural boundary: live web APIs vs. local artifacts**  
   The scrape pipeline does not rely on offline dumps. It queries live Wikipedia and Wikidata services through the API client layer in `wikiambig/api_clients/`. Everything fetched from the web is checkpointed under `output/scrape_data/`, then later assembled into the raw dataset under `output/raw_dataset/`. That boundary is important for the diagram: external data sources feed a resumable local pipeline.

3. **Entry points and orchestration**  
   The package exposes a CLI in `wikiambig/cli.py` with commands like `scrape` and `build`. Configuration is loaded from YAML via `wikiambig/config.py`, and progress/resume state is managed with `wikiambig/checkpoint.py`. The pipeline is designed so each stage can run independently and resume from atomically written intermediate files.

4. **Core scrape pipeline: seven sequential stages**  
   The main workflow lives in `wikiambig/pipeline/` and is explicitly staged:  
   - `s1_disam_index.py`: build a Wikipedia disambiguation-page index  
   - `s2_wikitext.py`: extract entity links from wikitext  
   - `s3_wikidata.py`: enrich QIDs with Wikidata metadata and coarse types  
   - `s4_wikipedia.py`: fetch article intros and image lists  
   - `s5_image_data.py`: resolve image metadata and usage counts  
   - `s6_visual_entity_data.py`: enrich entities referenced by shared images  
   - `s7_assemble.py`: assemble the final dataset and KB  
   This stage structure is the backbone of the architecture diagram.

5. **The key data model and where it is enforced**  
   `wikiambig/models.py` defines the repository’s shared entity shapes, such as mention entries, entity KB records, image metadata, and manifest-like structures. These models are the contract between stages, the assembler, and downstream split/eval tooling. The central instance concept is a `(mention, image)` pair whose answer is the unique QID in the intersection of textual and visual candidate sets.

6. **How the raw dataset is constructed**  
   The assembled output in `output/raw_dataset/` consists of `dataset.jsonl`, `entity_kb.json`, and `manifest.json`. Conceptually, S1–S6 build two candidate spaces: one from text surface forms and one from image reuse across articles. S7 merges those sources into a full, unfiltered dataset snapshot and records provenance/timestamp information in the manifest.

7. **Filtering and split generation as a separate downstream subsystem**  
   `split_gen/make_split.py` turns the raw assembled dataset into a benchmark split under `output/split_<name>/`, producing `instances.jsonl` and `kb.jsonl`. The split config in `configs/split_gen/` applies the quality filters described in the README: mention constraints, entity-type constraints, image constraints, and the uniqueness requirement that the text/visual candidate intersection contains exactly one answer.

8. **Inspection and analysis tooling**  
   `split_gen/view_split.py` is a human inspection tool that generates a self-contained HTML viewer for browsing instances. `fig_gen/` contains analysis scripts (`stats.py`, `stats_extended.py`, `mention_wordcloud.py`) that consume split files to produce dataset statistics and publication figures. These are read-only consumers of the benchmark outputs.

9. **Evaluation subsystem**  
   `eval/` is isolated from the data-building pipeline and focuses on running models against a split. `eval/run_eval.py` loads experiment YAML from `configs/eval/`, prompts from `eval/prompts/`, and model wrappers from `eval/models/` such as `qwen3.py`. This boundary matters because evaluation depends on the split format, not on the scrape internals.

10. **External service dependencies that shape the architecture**  
   The major external dependencies are Wikipedia and Wikidata: Wikipedia disambiguation pages and article wikitext provide mention/link structure, while Wikidata provides entity metadata, types, and URLs. Wikipedia image metadata and article image reuse are central to the visual-candidate side of the task. The whole dataset is therefore a live-source snapshot rather than a static corpus.

11. **Operational characteristics**  
   The pipeline is parallelized and rate-limited via config values such as worker count, batch sizes, and request pacing in `configs/scrape/default.yaml`. It writes intermediate checkpoints frequently so stages can be restarted without recomputing everything. `oar.sh` and `oar_eval.sh` indicate batch/HPC-oriented execution support.

12. **What an architecture diagram should emphasize**  
   The main flow is: Wikipedia/Wikidata APIs → staged scraper (`wikiambig/pipeline/`) → raw dataset artifacts → split generation and filtering → analysis/evaluation. The important subsystem boundary is between candidate extraction/enrichment and final benchmark curation. The final task is a multimodal disambiguation problem defined by the intersection of textual and visual candidate sets.

Main entry points: [`wikiambig/api_clients`](https://github.com/to2002td-cpu/dataset_builder_mel/tree/main/wikiambig/api_clients) (API clients), [`wikiambig/checkpoint.py`](https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/checkpoint.py) (Checkpointing), [`wikiambig/models.py`](https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/models.py) (Data models), [`wikiambig/pipeline/s1_disam_index.py`](https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/pipeline/s1_disam_index.py) (Disam index)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_external["External sources"]
  node_wiki_api(("Wikipedia API<br/>external source"))
  node_wd_api(("Wikidata API<br/>external source"))
end

subgraph group_core["wikiambig"]
  node_cli["CLI<br/>entrypoint<br/>[cli.py]"]
  node_config["Config<br/>[config.py]"]
  node_checkpoint["Checkpointing<br/>resume state<br/>[checkpoint.py]"]
  node_models[("Data models<br/>schema layer<br/>[models.py]")]
  node_api_clients["API clients<br/>service adapters"]
  node_s1["Disam index<br/>pipeline stage<br/>[s1_disam_index.py]"]
  node_s2["Wikitext links<br/>pipeline stage<br/>[s2_wikitext.py]"]
  node_s3["Wikidata enrich<br/>pipeline stage<br/>[s3_wikidata.py]"]
  node_s4["Article intros<br/>pipeline stage<br/>[s4_wikipedia.py]"]
  node_s5["Image data<br/>pipeline stage<br/>[s5_image_data.py]"]
  node_s6["Visual entities<br/>pipeline stage"]
  node_s7["Assemble raw<br/>pipeline stage<br/>[s7_assemble.py]"]
  node_raw_out[("Raw dataset<br/>artifact bundle")]
end

subgraph group_downstream["Consumers"]
  node_split_gen["Split gen<br/>curation tool<br/>[make_split.py]"]
  node_split_view["Split viewer<br/>inspection tool<br/>[view_split.py]"]
  node_fig_gen["Figure gen<br/>analysis tools<br/>[stats.py]"]
  node_eval["Evaluation<br/>model eval<br/>[run_eval.py]"]
end

node_wiki_api -->|"requests"| node_api_clients
node_wd_api -->|"requests"| node_api_clients
node_config -->|"loads"| node_cli
node_config -->|"drives"| node_s1
node_checkpoint -.->|"resumes"| node_s1
node_checkpoint -.->|"resumes"| node_s2
node_checkpoint -.->|"resumes"| node_s3
node_checkpoint -.->|"resumes"| node_s4
node_checkpoint -.->|"resumes"| node_s5
node_checkpoint -.->|"resumes"| node_s6
node_checkpoint -.->|"resumes"| node_s7
node_api_clients -->|"feeds"| node_s1
node_api_clients -->|"feeds"| node_s2
node_api_clients -->|"feeds"| node_s3
node_api_clients -->|"feeds"| node_s4
node_api_clients -->|"feeds"| node_s5
node_api_clients -->|"feeds"| node_s6
node_s1 -->|"pipeline"| node_s2
node_s2 -->|"pipeline"| node_s3
node_s3 -->|"pipeline"| node_s4
node_s4 -->|"pipeline"| node_s5
node_s5 -->|"pipeline"| node_s6
node_s6 -->|"pipeline"| node_s7
node_models -.->|"contracts"| node_s1
node_models -.->|"contracts"| node_s2
node_models -.->|"contracts"| node_s3
node_models -.->|"contracts"| node_s4
node_models -.->|"contracts"| node_s5
node_models -.->|"contracts"| node_s6
node_models -.->|"contracts"| node_s7
node_s7 -->|"writes"| node_raw_out
node_raw_out -->|"inputs"| node_split_gen
node_split_gen -->|"exports"| node_split_view
node_split_gen -->|"feeds"| node_fig_gen
node_split_gen -->|"feeds"| node_eval

click node_cli "https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/cli.py"
click node_config "https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/config.py"
click node_checkpoint "https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/checkpoint.py"
click node_models "https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/models.py"
click node_api_clients "https://github.com/to2002td-cpu/dataset_builder_mel/tree/main/wikiambig/api_clients"
click node_s1 "https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/pipeline/s1_disam_index.py"
click node_s2 "https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/pipeline/s2_wikitext.py"
click node_s3 "https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/pipeline/s3_wikidata.py"
click node_s4 "https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/pipeline/s4_wikipedia.py"
click node_s5 "https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/pipeline/s5_image_data.py"
click node_s6 "https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/pipeline/s6_visual_entity_data.py"
click node_s7 "https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/pipeline/s7_assemble.py"
click node_split_gen "https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/split_gen/make_split.py"
click node_split_view "https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/split_gen/view_split.py"
click node_fig_gen "https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/fig_gen/stats.py"
click node_eval "https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/eval/run_eval.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_wiki_api,node_wd_api toneBlue
class node_cli,node_config,node_checkpoint,node_models,node_api_clients,node_s1,node_s2,node_s3,node_s4,node_s5,node_s6,node_s7,node_raw_out toneAmber
class node_split_gen,node_split_view,node_fig_gen,node_eval toneMint
```

## Components

### External sources

Live web services queried by the scraping pipeline.

- **Wikipedia API**: external source · Live Wikipedia article, disambiguation, and image data source.
- **Wikidata API**: external source · Live Wikidata entity metadata, types, and URLs.

### wikiambig

Core package for configuration, checkpointing, API access, and staged dataset construction.

- [CLI](https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/cli.py): entrypoint · Command-line entry points for scrape and build workflows. · `wikiambig/cli.py`
- [Config](https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/config.py): yaml config · Loads YAML configuration for scrape, split, and eval runs. · `wikiambig/config.py`
- [Checkpointing](https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/checkpoint.py): resume state · Atomic intermediate-state tracking for restartable stages. · `wikiambig/checkpoint.py`
- [Data models](https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/models.py): schema layer · Shared mention, entity, image, and manifest structures used across stages. · `wikiambig/models.py`
- [API clients](https://github.com/to2002td-cpu/dataset_builder_mel/tree/main/wikiambig/api_clients): service adapters · Wrapper layer for Wikipedia and Wikidata requests. · `wikiambig/api_clients`
- [Disam index](https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/pipeline/s1_disam_index.py): pipeline stage · Builds the Wikipedia disambiguation-page index. · `wikiambig/pipeline/s1_disam_index.py`
- [Wikitext links](https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/pipeline/s2_wikitext.py): pipeline stage · Extracts entity links and mention candidates from Wikipedia wikitext. · `wikiambig/pipeline/s2_wikitext.py`
- [Wikidata enrich](https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/pipeline/s3_wikidata.py): pipeline stage · Adds Wikidata metadata and coarse entity types for QIDs. · `wikiambig/pipeline/s3_wikidata.py`
- [Article intros](https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/pipeline/s4_wikipedia.py): pipeline stage · Fetches article intros and image lists from Wikipedia. · `wikiambig/pipeline/s4_wikipedia.py`
- [Image data](https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/pipeline/s5_image_data.py): pipeline stage · Resolves image metadata and usage counts. · `wikiambig/pipeline/s5_image_data.py`
- [Visual entities](https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/pipeline/s6_visual_entity_data.py): pipeline stage · Enriches entities referenced by shared images. · `wikiambig/pipeline/s6_visual_entity_data.py`
- [Assemble raw](https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/wikiambig/pipeline/s7_assemble.py): pipeline stage · Merges candidate spaces into the raw dataset and KB snapshot. · `wikiambig/pipeline/s7_assemble.py`
- **Raw dataset**: artifact bundle · Assembled dataset.jsonl, entity_kb.json, and manifest.json.

### Consumers

Tools that curate, inspect, analyze, and evaluate the assembled benchmark.

- [Split gen](https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/split_gen/make_split.py): curation tool · Filters the raw dataset into a benchmark split. · `split_gen/make_split.py`
- [Split viewer](https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/split_gen/view_split.py): inspection tool · Creates a self-contained HTML browser for split inspection. · `split_gen/view_split.py`
- [Figure gen](https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/fig_gen/stats.py): analysis tools · Computes statistics and publication figures from split files. · `fig_gen/stats.py`
- [Evaluation](https://github.com/to2002td-cpu/dataset_builder_mel/blob/main/eval/run_eval.py): model eval · Runs models against a benchmark split using prompts and model wrappers. · `eval/run_eval.py`

## Connections

- Wikipedia API → API clients: requests · Wikipedia data is fetched through the client layer.
- Wikidata API → API clients: requests · Wikidata data is fetched through the client layer.
- Config → CLI: loads · CLI commands use YAML configuration.
- Config → Disam index: drives · Scrape stages read pipeline configuration.
- Checkpointing → Disam index: resumes · Stages read and write checkpoint state.
- Checkpointing → Wikitext links: resumes · Stages read and write checkpoint state.
- Checkpointing → Wikidata enrich: resumes · Stages read and write checkpoint state.
- Checkpointing → Article intros: resumes · Stages read and write checkpoint state.
- Checkpointing → Image data: resumes · Stages read and write checkpoint state.
- Checkpointing → Visual entities: resumes · Stages read and write checkpoint state.
- Checkpointing → Assemble raw: resumes · Stages read and write checkpoint state.
- API clients → Disam index: feeds · Client outputs support disambiguation indexing.
- API clients → Wikitext links: feeds · Client outputs support wikitext extraction.
- API clients → Wikidata enrich: feeds · Client outputs support Wikidata enrichment.
- API clients → Article intros: feeds · Client outputs support article and image retrieval.
- API clients → Image data: feeds · Client outputs support image metadata resolution.
- API clients → Visual entities: feeds · Client outputs support visual-entity enrichment.
- Disam index → Wikitext links: pipeline · Later stages consume disambiguation structure.
- Wikitext links → Wikidata enrich: pipeline · Text-derived candidates are enriched with Wikidata data.
- Wikidata enrich → Article intros: pipeline · Entity metadata informs article intros and image lookups.
- Article intros → Image data: pipeline · Image lists are expanded into image metadata and usage counts.
- Image data → Visual entities: pipeline · Image usage drives shared-image entity enrichment.
- Visual entities → Assemble raw: pipeline · Visual candidate data flows into final assembly.
- Data models → Disam index: contracts · Shared structures define stage inputs and outputs.
- Data models → Wikitext links: contracts · Shared structures define stage inputs and outputs.
- Data models → Wikidata enrich: contracts · Shared structures define stage inputs and outputs.
- Data models → Article intros: contracts · Shared structures define stage inputs and outputs.
- Data models → Image data: contracts · Shared structures define stage inputs and outputs.
- Data models → Visual entities: contracts · Shared structures define stage inputs and outputs.
- Data models → Assemble raw: contracts · Shared structures define stage inputs and outputs.
- Assemble raw → Raw dataset: writes · The assembler materializes the raw dataset snapshot.
- Raw dataset → Split gen: inputs · Split generation consumes the assembled raw dataset.
- Split gen → Split viewer: exports · Viewer reads generated split artifacts.
- Split gen → Figure gen: feeds · Statistics and figures are computed from split outputs.
- Split gen → Evaluation: feeds · Evaluation runs on split artifacts and prompts.