to2002td-cpu/dataset_builder_mel

1. Repo purpose and top-level shape

This repository is a data-building pipeline for a multimodal named entity disambiguation benchmark. It is organized as a multi-stage ETL system plus downstream analysis/evaluation tools. The core package is wikiambig/; the other top-level directories are consumers of its outputs: split_gen/ for filtering a final split, fig_gen/ for statistics/figures, and eval/ for model evaluation.

2. Primary architectural boundary: live web APIs vs. local artifacts

The scrape pipeline does not rely on offline dumps. It queries live Wikipedia and Wikidata services through the API client layer in wikiambig/api_clients/. Everything fetched from the web is checkpointed under output/scrape_data/, then later assembled into the raw dataset under output/raw_dataset/. That boundary is important for the diagram: external data sources feed a resumable local pipeline.

3. Entry points and orchestration

The package exposes a CLI in wikiambig/cli.py with commands like scrape and build. Configuration is loaded from YAML via wikiambig/config.py, and progress/resume state is managed with wikiambig/checkpoint.py. The pipeline is designed so each stage can run independently and resume from atomically written intermediate files.

4. Core scrape pipeline: seven sequential stages

The main workflow lives in wikiambig/pipeline/ and is explicitly staged:

- s1_disam_index.py: build a Wikipedia disambiguation-page index

- s2_wikitext.py: extract entity links from wikitext

- s3_wikidata.py: enrich QIDs with Wikidata metadata and coarse types

- s4_wikipedia.py: fetch article intros and image lists

- s5_image_data.py: resolve image metadata and usage counts

- s6_visual_entity_data.py: enrich entities referenced by shared images

- s7_assemble.py: assemble the final dataset and KB

This stage structure is the backbone of the architecture diagram.

5. The key data model and where it is enforced

wikiambig/models.py defines the repository’s shared entity shapes, such as mention entries, entity KB records, image metadata, and manifest-like structures. These models are the contract between stages, the assembler, and downstream split/eval tooling. The central instance concept is a (mention, image) pair whose answer is the unique QID in the intersection of textual and visual candidate sets.

6. How the raw dataset is constructed

The assembled output in output/raw_dataset/ consists of dataset.jsonl, entity_kb.json, and manifest.json. Conceptually, S1–S6 build two candidate spaces: one from text surface forms and one from image reuse across articles. S7 merges those sources into a full, unfiltered dataset snapshot and records provenance/timestamp information in the manifest.

7. Filtering and split generation as a separate downstream subsystem

split_gen/make_split.py turns the raw assembled dataset into a benchmark split under output/split_<name>/, producing instances.jsonl and kb.jsonl. The split config in configs/split_gen/ applies the quality filters described in the README: mention constraints, entity-type constraints, image constraints, and the uniqueness requirement that the text/visual candidate intersection contains exactly one answer.

8. Inspection and analysis tooling

split_gen/view_split.py is a human inspection tool that generates a self-contained HTML viewer for browsing instances. fig_gen/ contains analysis scripts (stats.py, stats_extended.py, mention_wordcloud.py) that consume split files to produce dataset statistics and publication figures. These are read-only consumers of the benchmark outputs.

9. Evaluation subsystem

eval/ is isolated from the data-building pipeline and focuses on running models against a split. eval/run_eval.py loads experiment YAML from configs/eval/, prompts from eval/prompts/, and model wrappers from eval/models/ such as qwen3.py. This boundary matters because evaluation depends on the split format, not on the scrape internals.

10. External service dependencies that shape the architecture

The major external dependencies are Wikipedia and Wikidata: Wikipedia disambiguation pages and article wikitext provide mention/link structure, while Wikidata provides entity metadata, types, and URLs. Wikipedia image metadata and article image reuse are central to the visual-candidate side of the task. The whole dataset is therefore a live-source snapshot rather than a static corpus.

11. Operational characteristics

The pipeline is parallelized and rate-limited via config values such as worker count, batch sizes, and request pacing in configs/scrape/default.yaml. It writes intermediate checkpoints frequently so stages can be restarted without recomputing everything. oar.sh and oar_eval.sh indicate batch/HPC-oriented execution support.

12. What an architecture diagram should emphasize

The main flow is: Wikipedia/Wikidata APIs → staged scraper (wikiambig/pipeline/) → raw dataset artifacts → split generation and filtering → analysis/evaluation. The important subsystem boundary is between candidate extraction/enrichment and final benchmark curation. The final task is a multimodal disambiguation problem defined by the intersection of textual and visual candidate sets.