How tascj/kaggle-lmsys-chatbot-arena works
Main Components
• Data Preparation & Artifacts:
• External data (CSV files, training/test splits) is intended to be located in a separate “data” directory (outside the repo) and “artifacts” where processed output is stored.
• scripts/prepare_dataset.py kick-starts the preparation stage.
• Multi-Stage Training Pipeline:
• Stage 1: Represented by configs/stage1. This stage contains initial training scripts (m0.py, m1.py, m3.py).
• Pseudo-label Generation: Located in configs/stage1_generate_pseudo_labels. This stage has variants of training scripts (m0_110k.py, m0_130k.py, etc.) that likely generate additional labels or training data adjustments.
• Stage 2: Found in configs/stage2 with its own README and scripts (e.g., m0.py, m3.py).
• Stage 3: Located in configs/stage3 and includes both training and validation scripts (m0.py, m0val.py, m3.py, m3val.py).
• Human Preference Module:
• The human_pref directory handles model optimizers, data collation, dataset preparation, processors, and both training/inference logic specific to human preference predictions.
• It contains inference submodules for models (modeling_gemma2.py, modeling_llama.py), various custom operations for performance optimizations (flash_attention, fused rotary embedding, etc.), and utility functions.
• Orchestration & Execution:
• main.py acts as the entry point to the system (likely coordinating the overall training/inference flow).
• The scripts subdirectory contains additional helper scripts:
▪ scripts/kapenon and scripts/sakaku serve as parts of the training or evaluation workflows.
▪ Scripts for converting checkpoints for submission (prepare_gemma2_for_submission.py, prepare_llama3_for_submission.py).
Mapping the Relationships and Interactions
• Data flows from raw CSV files (data) into the dataset preparation script (scripts/prepare_dataset.py).
• Once prepared, training proceeds via a series of stages:
• Stage 1 training (configs/stage1) produces initial model checkpoints.
• Pseudo-label generation (configs/stage1_generate_pseudo_labels) enhances or refines these predictions.
• Stage 2 (configs/stage2) further refines the model training.
• Stage 3 (configs/stage3) finalizes training and validation.
• The human_pref module provides shared components (data pipelines, model optimizations, custom ops) that are integrated throughout these stages.
• After training, conversion scripts under the scripts folder adjust the model checkpoints for inference and submission (pointing to external Kaggle requirements).
Important Architectural Patterns and Design Principles
• Pipeline and Modularization:
• The clear segmentation into multiple training stages (stage1, pseudo-label generation, stage2, stage3) shows a staged pipeline design.
• Separation of concerns: Different folders for configuration, human preference specifics, and orchestration (main.py, scripts).
• Reusability and Extensibility:
• The human_pref directory encapsulates data processing and model inference functionalities separately to allow possible reuse or extension.
• Dependency on External Libraries:
• Critical frameworks include PyTorch (via the NVIDIA container), detectron2, transformers, datasets, and flash-attn which are integrated across various components.
Final Element Suggestions
• Data ingestion → Training Stages → Inference conversion.
• Optionally, outline an area for external services or hardware dependencies (e.g., GPU configuration and Docker image nvcr.io/nvidia/pytorch:24.04-py3).