# nguyenducanh20235585/ielts-data-pipeline architecture

> Architecture diagram of the GitHub repository nguyenducanh20235585/ielts-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nguyenducanh20235585/ielts-data-pipeline
- Repository: https://github.com/nguyenducanh20235585/ielts-data-pipeline

## Overview

This repository builds IELTS Reading datasets from PDFs, from source extraction and question-candidate generation through human review, answer mapping, QA, finalization, and optional import. It also contains universal and Cambridge-specific importers and validation/finalization paths. The README is empty; edges are limited to relationships visible in excerpts or the SOURCE INDEX, so the graph does not infer a complete stage sequence from filenames or shared conventions.

Main entry points: [`stage3_review.json`](https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/stage3_review.json) (Review instructions), [`pipeline/extract_pdf_layout.py`](https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/pipeline/extract_pdf_layout.py) (PDF layout extraction), [`pipeline/prepare_reading_source.py`](https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/pipeline/prepare_reading_source.py) (Source preparation), [`pipeline/build_reading_candidates.py`](https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/pipeline/build_reading_candidates.py) (Candidate building)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_source["Source processing"]
  node_extract["PDF layout extraction"]
  node_prepare["Source preparation"]
  node_structure["Structure detection"]
  node_runner["Pipeline runner<br/>[run_pipeline.py]"]
end

subgraph group_content["Question content"]
  node_parse["Question parsing"]
  node_candidates["Candidate building"]
  node_answers["Answer-key mapping<br/>[map_answer_keys.py]"]
  node_universalanswers["Universal answer mapping"]
end

subgraph group_quality["Review and QA"]
  node_reviews["Review application"]
  node_qa["Dataset validation"]
  node_universalqa["Universal content QA"]
end

subgraph group_delivery["Dataset delivery"]
  node_finalize["Dataset finalization"]
  node_universalfinalize["Universal finalization"]
  node_importer["Cambridge importer"]
  node_ieltsimport["IELTS importer<br/>[import_ielts.py]"]
end

node_pdf["Reading PDF"]
node_human(("Content reviewer"))
node_reviewdata["Review instructions<br/>[stage3_review.json]"]
node_database[("Content database")]

node_pdf -->|"extracts layout"| node_extract
node_prepare -->|"reuses extractor"| node_extract
node_candidates -->|"reuses parser"| node_parse
node_human -->|"supplies review"| node_reviewdata
node_reviewdata -->|"applies patches"| node_reviews
node_importer -.->|"writes drafts"| node_database

click node_extract "https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/pipeline/extract_pdf_layout.py"
click node_prepare "https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/pipeline/prepare_reading_source.py"
click node_structure "https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/pipeline/detect_reading_structure.py"
click node_parse "https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/pipeline/parse_reading_questions.py"
click node_candidates "https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/pipeline/build_reading_candidates.py"
click node_reviews "https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/review/apply_question_reviews.py"
click node_reviewdata "https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/stage3_review.json"
click node_answers "https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/pipeline/map_answer_keys.py"
click node_universalanswers "https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/pipeline/map_reading_answers_universal.py"
click node_qa "https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/review/validate_reading_dataset.py"
click node_universalqa "https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/review/qa_reading_universal.py"
click node_finalize "https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/review/finalize_reading_dataset.py"
click node_universalfinalize "https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/review/finalize_reading_universal.py"
click node_importer "https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/importers/import_cambridge21_reading_test1.py"
click node_ieltsimport "https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/importers/import_ielts.py"
click node_runner "https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/run_pipeline.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_extract,node_prepare,node_structure,node_runner,node_human,node_reviewdata toneBlue
class node_parse,node_candidates,node_answers,node_universalanswers,node_database toneAmber
class node_reviews,node_qa,node_universalqa toneMint
class node_finalize,node_universalfinalize,node_importer,node_ieltsimport toneRose
class node_pdf toneIndigo
```

## Components

### Source processing

- [PDF layout extraction](https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/pipeline/extract_pdf_layout.py): `pipeline/extract_pdf_layout.py`
- [Source preparation](https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/pipeline/prepare_reading_source.py): `pipeline/prepare_reading_source.py`
- [Structure detection](https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/pipeline/detect_reading_structure.py): `pipeline/detect_reading_structure.py`
- [Pipeline runner](https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/run_pipeline.py): `run_pipeline.py`

### Question content

- [Question parsing](https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/pipeline/parse_reading_questions.py): `pipeline/parse_reading_questions.py`
- [Candidate building](https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/pipeline/build_reading_candidates.py): `pipeline/build_reading_candidates.py`
- [Answer-key mapping](https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/pipeline/map_answer_keys.py): `pipeline/map_answer_keys.py`
- [Universal answer mapping](https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/pipeline/map_reading_answers_universal.py): `pipeline/map_reading_answers_universal.py`

### Review and QA

- [Review application](https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/review/apply_question_reviews.py): `review/apply_question_reviews.py`
- [Dataset validation](https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/review/validate_reading_dataset.py): `review/validate_reading_dataset.py`
- [Universal content QA](https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/review/qa_reading_universal.py): `review/qa_reading_universal.py`

### Dataset delivery

- [Dataset finalization](https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/review/finalize_reading_dataset.py): `review/finalize_reading_dataset.py`
- [Universal finalization](https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/review/finalize_reading_universal.py): `review/finalize_reading_universal.py`
- [Cambridge importer](https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/importers/import_cambridge21_reading_test1.py): `importers/import_cambridge21_reading_test1.py`
- [IELTS importer](https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/importers/import_ielts.py): `importers/import_ielts.py`

### Other components

- **Reading PDF**
- **Content reviewer**
- [Review instructions](https://github.com/nguyenducanh20235585/ielts-data-pipeline/blob/master/stage3_review.json): `stage3_review.json`
- **Content database**

## Connections

- Reading PDF → PDF layout extraction: extracts layout (evidence: `pipeline/extract_pdf_layout.py`)
- Source preparation → PDF layout extraction: reuses extractor (evidence: `pipeline/prepare_reading_source.py`)
- Candidate building → Question parsing: reuses parser (evidence: `pipeline/build_reading_candidates.py`)
- Content reviewer → Review instructions: supplies review
- Review instructions → Review application: applies patches (evidence: `review/apply_question_reviews.py`)
- Cambridge importer → Content database: writes drafts