How hanmoco/cefr-mcq-simulator works
CEFR MCQ Learner Simulator converts Cambridge reading-question data into examples with learner response distributions, then uses those examples for zero-shot evaluation or fine-tuning a language model. Its principal workflow is raw data → preprocessing → optional synthetic augmentation → training/evaluation with passage-level splits → saved outputs. The README documents these workflows; excerpts show the data conversion and synthetic-generation logic and parts of both model runners, but not their complete orchestration.
Main entry points: train.py Fine-tuning · zeroshot.py Zero-shot evaluation · data Processed examples · split_ft.json Passage splits
Components 11
Data preparation
- Dataset conversion
src/generate_dataset.py - Synthetic augmentation
src/generate_synthetic_dataset.py
Model runs
- Zero-shot evaluation
src/zeroshot.py - Fine-tuning
src/train.py
Inputs and outputs
- Processed examples
data - Synthetic examples
data - Passage splits
splits/split_ft.json - Run outputs
data
Other components
- Researcher
- Cambridge dataset
- Pretrained language model
Connections 16
- Researcher → Cambridge dataset: obtains data
README.md - Cambridge dataset → Dataset conversion: provides JSONL
README.md - Dataset conversion → Processed examples: writes examples
src/generate_dataset.py - Processed examples → Synthetic augmentation: provides input
README.md - Synthetic augmentation → Synthetic examples: writes augmented data
src/generate_synthetic_dataset.py - Researcher → Zero-shot evaluation: runs evaluation
README.md - Researcher → Fine-tuning: runs fine-tuning
README.md - Processed examples → Zero-shot evaluation: supplies examples
README.md - Processed examples → Fine-tuning: supplies examples
README.md - Synthetic examples → Fine-tuning: supplies augmented examples
README.md - Passage splits → Zero-shot evaluation: selects passages
README.md - Passage splits → Fine-tuning: selects passages
README.md - Zero-shot evaluation → Pretrained language model: loads model
src/zeroshot.py - Fine-tuning → Pretrained language model: loads and fine-tunes
src/train.py - Zero-shot evaluation → Run outputs: writes results
README.md - Fine-tuning → Run outputs: writes results
README.md