How dvlab-research/motcoder works
Project type and purpose
• MoTCoder is a research codebase for instruction‐tuning large language models (LLMs) with a “Modular of Thought” framework.
• It consists of two main pipelines:
• Training (using LLaMA-Factory + YAML configs)
• Evaluation (using Hugging Face datasets, vLLM wrappers, custom scripts)
Main components
A. External Resources / Data Stores
• HF Hub Datasets: APPS, CodeContests
• HF Hub Models: MoTCoder‐7B, ‐32B, ‐72B
• Local “motcode” dataset (350K examples)
B. Training Pipeline
• LLaMA-Factory framework (cloned from GitHub)
• train/configs/*.yaml (hyperparams & model‐specific settings)
• train/scripts/*.sh (entrypoints: run.sh, run_1gpu.sh, etc.)
• Compute Layer: GPU cluster or single‐GPU node
• Outputs: fine-tuned model checkpoints
C. Evaluation Pipeline
• eval/dataset (data prep: data.py, prompts.py)
• eval/eval.py (evaluation harness)
• eval/vllm_gen.py & eval/vllm_fix.py (vLLM integration for generation & self‐correction)
• eval/scripts/*.sh (batch evaluation for APPS, CodeContests, fix tasks)
• eval/testing_util.py & eval/utils.py (metric computation & logging)
• Outputs: performance metrics, pass@1 results
D. Supporting Files
• .gitignore, Readme.md, imgs/ (documentation and visuals)
Relationships and data flows
• Data Flow for Training:
1. “motcode” dataset (HF Hub) → LLaMA-Factory data loader
3. LLaMA-Factory trains → GPU cluster → produces model checkpoints → optionally pushed to HF Hub
• Data Flow for Evaluation:
1. Download APPS & CodeContests → local data/ directory
2. eval/dataset/data.py transforms raw data → standardized prompts
3. eval scripts invoke eval/eval.py → load MoTCoder checkpoint
4. vLLM wrappers generate predictions + self‐corrections
5. utils/testing_util.py computes pass@1 and other metrics → logs/results
Architectural patterns & design principles
• Pipeline separation: clear division between training vs. evaluation
• Configuration‐driven: YAML configs decouple code from hyperparameters
• Modular code: distinct Python modules for data prep, generation, fixing, and utilities
• Reproducibility: shell scripts capture standard commands for various GPU setups