How wahab4114/dibert works
Project Type and Purpose
• This is an NLP research / training pipeline for “DIBERT,” a BERT-Base variant with an extra Parent Prediction (PP) objective.
• It comprises two major phases:
• Pretraining (on WikiText-103 with MLM, NSP, PP objectives)
• Downstream Finetuning & Evaluation (on tasks like QNLI, MRPC, SST-2, IMDB, SciTail, LIAR)
Directory Structure & Architectural Patterns
A. Top-level components
• config/ ← JSON configs for pretrain/finetune runs
• dibert.yml ← environment/dependency spec
• datapreparation.py, preprocessing.py
• dataset.py ← dataset abstraction for WikiText-103
• model.py ← DIBERT model architecture (MLM + NSP + PP heads)
• train.py ← pretraining orchestrator
• test_for_paper.py ← aggregated experiment runner / evaluation script
• utils.py ← shared helper functions
• results/ ← pretraining result artifacts (models, logs)
• Downstreamtask/ ← one subfolder per finetuning task
– Each subfolder has:
· <task>config.py
· <task>dataset.py
· <task>model.py
· train_<task>.py
· test_<task>.py
· results/ ← finetuning outcome (params, logs, seeds)
B. Architectural patterns
• Modular pipeline: clean separation of data prep, model definition, training loops, and evaluation.
• Configuration-driven orchestration: JSON/YAML configs injected into scripts.
• Experiment tracking via filesystem hierarchy (results/…).
Key Insights from README
• model.py: core DIBERT network (BERT + PP head)
• datapreparation.py + dataset.py: building MLM, NSP, PP labels
• Downstreamtask/**: mirrored pipeline per task, finetuned on both BERT & DIBERT
• dibert.yml: reproducible environment
• utils.py: shared across both pretraining and finetuning