# bigcode-project/astraios architecture

> Architecture diagram of the GitHub repository bigcode-project/astraios, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bigcode-project/astraios
- Repository: https://github.com/bigcode-project/astraios

## Overview

##### Project Type and Purpose
- Astraios is a research‐oriented toolkit and pipeline for parameter‐efficient instruction tuning of code large language models (LLMs).
- It combines:
- A custom PEFT library (“peft/”) for various fine-tuning methods (LoRA, IA3, AdapterH/P, Parallel, FFT).
- Training scripts (“training/”) that invoke the PEFT library via Accelerate on GPU/CPU.
- Evaluation scripts (“evaluation/”) that integrate with the BigCode evaluation harness.
- Precomputed outputs (“outputs/”) and visuals (“visuals/”) for figures and analysis.

##### Core Components
a. PEFT Library
- Location: peft/src/peft
- Provides tuner implementations, config loader, save/load, helper utilities.
- Publishes a Python package with setup.py / pyproject.toml.
b. Training Module
- Location: training/finetune.py, training/peft_config.py, run_peft.sh, run_fft.sh
- Orchestrates model loading (StarCoderBase), applies PEFT adapters, calls Accelerate for distributed training.
c. Evaluation Module
- Location: evaluation/*.sh, evaluation/eval_code_comprehension.ipynb
- Clones and installs bigcode-evaluation-harness, runs tasks (clone detection, defect detection, HumanEval variants) via accelerate launch main.py.
d. Data & Model Repositories (External Services)
- Hugging Face datasets: CommitPackFT+OASST, BigCloneBench, Devign, HumanEvalPack, ReCode, Asleep At The Keyboard.
- Hugging Face model hubs: starcoderbase-*, astraios-* collections.
e. Outputs Storage
- Location: outputs/ – JSON files of generations per task/method.
f. Visualization Assets
- Location: visuals/ – banner.png and generated figures via Colab.
g. User & CI/CD Interface
- Command-line shell scripts for setup and execution.
- Documentation in README and peft/docs for publishing the library.

##### Relationships & Data Flow
1. Developer (or CI) checks out Astraios repo.
2. Installs dependencies: pip install -e peft/, pip install bigcode-evaluation-harness.
3. **Training Flow**:
- Training script (run_peft.sh / run_fft.sh) → loads base model from HF → applies PEFT library → runs Accelerate on local/cluster GPU(s) → produces fine-tuned adapters or FFT models → pushes artifacts to outputs/ and optionally HF hub.
4. **Evaluation Flow**:
- Evaluation scripts → clones bigcode-evaluation-harness → for each task, loads base or PEFT-tuned model → generates predictions → writes JSON generations to outputs/ → collects metrics via harness → logs results.
5. **Visualization Flow**:
- Notebooks (Colab) read outputs/ JSON → produce plots/figures → saved under visuals/ and published in paper.

##### Architectural Patterns & Design Principles
- Modular library (PEFT) with clear API separation between tuner implementations and model/application code.
- Script-driven orchestration (bash wrappers) for reproducibility.
- External integration points: Hugging Face (models, datasets), BigCode evaluation harness.
- Configuration via Python scripts and YAML (Accelerate, Dockerfiles).
- Separation of concerns: training, evaluation, outputs, documentation.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Entities
    DevCI["Developer / CI"]:::external
    HFData["Hugging Face Datasets"]:::external
    HFModels["Hugging Face Model Hub"]:::external
    BigCode["BigCode Evaluation Harness"]:::external
    Colab["Colab Notebook"]:::external

    %% Internal Components
    subgraph "PEFT Library"
        direction TB
        PLCore["peft/src/peft"]:::internal
        PLMeta["peft/setup.py & pyproject.toml"]:::internal
        PLDocs["peft/docs"]:::internal
        subgraph "Docker Environments"
            PLDockCPU["Docker CPU"]:::internal
            PLDockGPU["Docker GPU"]:::internal
        end
        subgraph "Utility Scripts"
            PLScripts1["log_reports.py"]:::internal
            PLScripts2["stale.py"]:::internal
        end
    end

    subgraph "Training Module"
        direction TB
        TrainingScripts["run_peft.sh & run_fft.sh"]:::internal
        TrainingCode1["finetune.py"]:::internal
        TrainingCode2["peft_config.py"]:::internal
        Compute["Compute Cluster (Accelerate)"]:::internal
    end

    subgraph "Evaluation Module"
        direction TB
        EvaluationScripts["evaluation/*.sh"]:::internal
        Notebook["eval_code_comprehension.ipynb"]:::internal
    end

    Outputs[(JSON Outputs)]:::storage
    Visuals["Visuals (Figures)"]:::storage

    %% Flows
    DevCI -->|checks out & configures| PLCore
    DevCI -->|runs| TrainingScripts
    DevCI -->|runs| EvaluationScripts

    HFData -->|provides data| TrainingScripts
    HFData -->|provides data| EvaluationScripts
    HFModels -->|provides base model| TrainingScripts
    HFModels -->|provides base model| EvaluationScripts

    TrainingScripts -->|uses| PLCore
    TrainingScripts -->|imports| TrainingCode1
    TrainingScripts -->|imports| TrainingCode2
    TrainingScripts -->|launches| Compute
    Compute -->|writes adapters| Outputs
    Compute -.->|optional push| HFModels

    EvaluationScripts -->|clones| BigCode
    EvaluationScripts -->|loads model| PLCore
    EvaluationScripts -->|uses| BigCode
    EvaluationScripts -->|launches| Compute
    Compute -->|writes eval JSON| Outputs

    Colab -->|reads| Outputs
    Colab -->|generates plots| Visuals

    %% Click Events
    click PLCore "https://github.com/bigcode-project/astraios/tree/main/peft/src/peft"
    click PLMeta "https://github.com/bigcode-project/astraios/blob/main/peft/setup.py"
    click PLMeta "https://github.com/bigcode-project/astraios/blob/main/peft/pyproject.toml"
    click PLDocs "https://github.com/bigcode-project/astraios/tree/main/peft/docs"
    click PLDockCPU "https://github.com/bigcode-project/astraios/tree/main/peft/docker/peft-cpu/Dockerfile"
    click PLDockGPU "https://github.com/bigcode-project/astraios/tree/main/peft/docker/peft-gpu/Dockerfile"
    click PLScripts1 "https://github.com/bigcode-project/astraios/blob/main/peft/scripts/log_reports.py"
    click PLScripts2 "https://github.com/bigcode-project/astraios/blob/main/peft/scripts/stale.py"
    click TrainingScripts "https://github.com/bigcode-project/astraios/blob/main/training/run_peft.sh"
    click TrainingScripts "https://github.com/bigcode-project/astraios/blob/main/training/run_fft.sh"
    click TrainingCode1 "https://github.com/bigcode-project/astraios/blob/main/training/finetune.py"
    click TrainingCode2 "https://github.com/bigcode-project/astraios/blob/main/training/peft_config.py"
    click Notebook "https://github.com/bigcode-project/astraios/blob/main/evaluation/eval_code_comprehension.ipynb"
    click Outputs "https://github.com/bigcode-project/astraios/tree/main/outputs"
    click Visuals "https://github.com/bigcode-project/astraios/tree/main/visuals"

    %% Styles
    classDef internal fill:#D0E6FF,stroke:#0277BD
    classDef external fill:#E8F5E9,stroke:#2E7D32
    classDef storage fill:#FFF9C4,stroke:#F9A825
```
