How nguyenducanh20235585/ielts-data-pipeline works
This repository builds IELTS Reading datasets from PDFs, from source extraction and question-candidate generation through human review, answer mapping, QA, finalization, and optional import. It also contains universal and Cambridge-specific importers and validation/finalization paths. The README is empty; edges are limited to relationships visible in excerpts or the SOURCE INDEX, so the graph does not infer a complete stage sequence from filenames or shared conventions.
Main entry points: stage3_review.json Review instructions · extract_pdf_layout.py PDF layout extraction · prepare_reading_source.py Source preparation · build_reading_candidates.py Candidate building
Components 19
Source processing
- PDF layout extraction
pipeline/extract_pdf_layout.py - Source preparation
pipeline/prepare_reading_source.py - Structure detection
pipeline/detect_reading_structure.py - Pipeline runner
run_pipeline.py
Question content
- Question parsing
pipeline/parse_reading_questions.py - Candidate building
pipeline/build_reading_candidates.py - Answer-key mapping
pipeline/map_answer_keys.py - Universal answer mapping
pipeline/map_reading_answers_universal.py
Review and QA
- Review application
review/apply_question_reviews.py - Dataset validation
review/validate_reading_dataset.py - Universal content QA
review/qa_reading_universal.py
Dataset delivery
- Dataset finalization
review/finalize_reading_dataset.py - Universal finalization
review/finalize_reading_universal.py - Cambridge importer
importers/import_cambridge21_reading_test1.py - IELTS importer
importers/import_ielts.py
Other components
- Reading PDF
- Content reviewer
- Review instructions
stage3_review.json - Content database
Connections 6
- Reading PDF → PDF layout extraction: extracts layout
pipeline/extract_pdf_layout.py - Source preparation → PDF layout extraction: reuses extractor
pipeline/prepare_reading_source.py - Candidate building → Question parsing: reuses parser
pipeline/build_reading_candidates.py - Content reviewer → Review instructions: supplies review
- Review instructions → Review application: applies patches
review/apply_question_reviews.py - Cambridge importer → Content database: writes drafts