How maxreciprocate/kaggle-lmarena-1st-place works
Project Type and Purpose
• This is a multi‑stage machine‑learning training and distillation pipeline built for the WSDM Cup Multilingual Chatbot Arena. It orchestrates three main stages—pretraining, teacher training, and student distillation—on an HPC cluster using SLURM and DeepSpeed, and culminates in packing the distilled student models for evaluation.
File Structure Analysis
• Top‑level scripts:
• run.sh, train.py, format.py, label.py — entry points and utility scripts.
• launch.slurm, label.slurm — SLURM job definitions that dispatch jobs to the cluster.
• Stage directories:
• stage1/ (prepare_pretrain_data.py)
• stage2/ (prepare_teacher_data.py)
• stage3/ (collect_labels.py, merge_students.py, pack_student.py, prepare_student_data.py, prepare_synth_data.py)
• Packing library:
• packing/data — dataset loading + collators/processors
• packing/models — model definitions (Gemma2, Llama, Qwen2) + custom ops for FlashAttention, fused rotary embeddings, etc.
• Configuration files: requirements.txt, models.py, utils.py, LICENSE, README.md
README Insights
• Describes three‑stage workflow: pretraining → teacher training → distillation.
• Lists external data sources: Kaggle challenge data, nbroad datasets, HuggingFace space.
• Defines hardware/software requirements: H100 GPUs, Python 3.12, SLURM, DeepSpeed, Accelerate, Transformers, Flash‑Attention.
Diagramming the System Architecture
a. Main Components
• Data Ingestion Layer: local data/ directory, Kaggle storage, HuggingFace datasets
• Pretraining Stage (stage1): prepare_pretrain_data.py → DeepSpeed pretraining jobs on cluster → checkpoints
• Teacher Training Stage (stage2): prepare_teacher_data.py → transformers-based teacher fine‑tuning → teacher model artifacts
• Distillation Stage (stage3):
– collect_labels.py (gather teacher outputs)
– merge_students.py (aggregate multiple student runs)
– prepare_student_data.py / prepare_synth_data.py (assemble distillation dataset)
– pack_student.py (serialize final student model)
• Packing Library: packing/ submodule handles dataset collators, model wrappers, and custom kernels
• Orchestration & Jobs: run.sh, launch.slurm, label.slurm, label.sh
• External Services: Kaggle API, HuggingFace Hub
• Compute Infrastructure: SLURM scheduler controlling multiple H100 GPUs via DeepSpeed
• Artifact Storage: ckpts/ for model checkpoints, logs, and final packaged models
b. Relationships and Data Flow
• Data flows from external sources → local data/ → stage1 prep → pretraining → ckpts → stage2 prep → teacher training → teacher ckpts → stage3 data prep → distillation → student ckpts → packaging → deliverable models.
• Orchestration scripts trigger each stage on SLURM; each training job interacts with the packing library for data/model I/O.
• Custom ops in packing/models/ops are linked into training code via PyTorch extensions.
c. Architectural Patterns and Design Principles
• Pipeline pattern: clearly separated stages connected by on‑disk checkpoints.
• Modularization: packing submodule abstracts data processing and model implementation.
• Scalability: use of DeepSpeed + SLURM for distributed training.
• Extensibility: new model architectures or data processors can be dropped into packing/models or packing/data.
d. Key Technologies & Frameworks
• Python 3.12, PyTorch, Transformers
• Deepspeed for ZeRO optimization
• Accelerate for cross‑node orchestration
• Flash‑Attention custom kernels via Triton
• SLURM for job scheduling
Diagramming Conventions
• Clear labels: name every script, library, data store, compute node.
• Rectangles for compute components (training jobs, orchestrators)