How bigcode-project/astraios works
Project Type and Purpose
• Astraios is a research‐oriented toolkit and pipeline for parameter‐efficient instruction tuning of code large language models (LLMs).
• It combines:
• A custom PEFT library (“peft/”) for various fine-tuning methods (LoRA, IA3, AdapterH/P, Parallel, FFT).
• Training scripts (“training/”) that invoke the PEFT library via Accelerate on GPU/CPU.
• Evaluation scripts (“evaluation/”) that integrate with the BigCode evaluation harness.
• Precomputed outputs (“outputs/”) and visuals (“visuals/”) for figures and analysis.
Core Components
a. PEFT Library
• Location: peft/src/peft
• Provides tuner implementations, config loader, save/load, helper utilities.
• Publishes a Python package with setup.py / pyproject.toml.
b. Training Module
• Location: training/finetune.py, training/peft_config.py, run_peft.sh, run_fft.sh
• Orchestrates model loading (StarCoderBase), applies PEFT adapters, calls Accelerate for distributed training.
c. Evaluation Module
• Location: evaluation/*.sh, evaluation/eval_code_comprehension.ipynb
• Clones and installs bigcode-evaluation-harness, runs tasks (clone detection, defect detection, HumanEval variants) via accelerate launch main.py.
d. Data & Model Repositories (External Services)
• Hugging Face datasets: CommitPackFT+OASST, BigCloneBench, Devign, HumanEvalPack, ReCode, Asleep At The Keyboard.
• Hugging Face model hubs: starcoderbase-*, astraios-* collections.
e. Outputs Storage
• Location: outputs/ – JSON files of generations per task/method.
f. Visualization Assets
• Location: visuals/ – banner.png and generated figures via Colab.
g. User & CI/CD Interface
• Command-line shell scripts for setup and execution.
• Documentation in README and peft/docs for publishing the library.
Relationships & Data Flow
1. Developer (or CI) checks out Astraios repo.
2. Installs dependencies: pip install -e peft/, pip install bigcode-evaluation-harness.
3. Training Flow:
• Training script (run_peft.sh / run_fft.sh) → loads base model from HF → applies PEFT library → runs Accelerate on local/cluster GPU(s) → produces fine-tuned adapters or FFT models → pushes artifacts to outputs/ and optionally HF hub.
4. Evaluation Flow:
• Evaluation scripts → clones bigcode-evaluation-harness → for each task, loads base or PEFT-tuned model → generates predictions → writes JSON generations to outputs/ → collects metrics via harness → logs results.
5. Visualization Flow:
• Notebooks (Colab) read outputs/ JSON → produce plots/figures → saved under visuals/ and published in paper.
Architectural Patterns & Design Principles
• Modular library (PEFT) with clear API separation between tuner implementations and model/application code.
• Script-driven orchestration (bash wrappers) for reproducibility.
• External integration points: Hugging Face (models, datasets), BigCode evaluation harness.
• Configuration via Python scripts and YAML (Accelerate, Dockerfiles).
• Separation of concerns: training, evaluation, outputs, documentation.