# bigcode-project/starcoder architecture

> Architecture diagram of the GitHub repository bigcode-project/starcoder, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bigcode-project/starcoder
- Repository: https://github.com/bigcode-project/starcoder

## Overview

##### Project type and purpose
- This repository is not a web application but an open-source toolkit for working with StarCoder, a large LM for code.
- It provides two main pipelines:
- “chat/”: code to fine-tune StarCoder as a conversational assistant and run inference (generate.py, train.py, dialogues.py, config files).
- “finetune/”: generic scripts to fine-tune on instruction datasets and to merge PEFT adapters.
- It relies on Hugging Face’s Transformers, Datasets, PEFT, BitsAndBytes, DeepSpeed, and optional experiment tracking (wandb).

##### File-structure analysis
- Top level:
- README.md, requirements.txt, licensing, `.gitignore`.
- `chat/`: scripts + configs for building a chat assistant (config.py, YAML/JSON, dialogues, generate, train, utils).
- `finetune/`: scripts for generic fine-tuning (finetune.py) and merging adapters (merge_peft_adapters.py).
- Each script is a CLI entrypoint that imports shared helper code (`chat/utils.py`) and external libraries.
- Config files drive both train and generate pipelines.
- Requirements files enumerate precise dependencies per component.

##### README insights
- Describes quickstart for code generation and inference.
- Shows how to run Docker TGI for serving.
- Outlines fine-tuning steps: conda environment, libraries’ versions, dataset sources, hyperparameters.
- Mentions merging adapters and pushing to the Hub, and hardware requirements.

a. Main components
1. CLI/Client layer
– chat/train.py (DL training loop)
– chat/generate.py (inference entrypoint)
– finetune/finetune.py (instruction-tuning pipeline)
– finetune/merge_peft_adapters.py (model post-processing)
2. Configuration layer
– chat/config.py / config.yaml / JSON Deepspeed config
– CLI flags in fine-tune scripts
3. Core libraries
– Transformers model loader (AutoModelForCausalLM, AutoTokenizer)
– PEFT adapters + BitsAndBytes for efficient finetuning & quantization
– Datasets streaming (Hugging Face Datasets)
– DeepSpeed (Z3, bf16 config)
4. External services
– Hugging Face Hub (model storage, token authentication)
– Experiment tracker (wandb)
5. Data stores
– Local or remote dataset (e.g., ArmelR/stack-exchange-instruction)
– Checkpoints directory (`./checkpoints`)

b. Relationships and interactions
- Fine-tuning pipeline (finetune.py) reads config & dataset → builds dataset loader → wraps model in PEFT → trains with accelerate/deepspeed → saves adapter checkpoints → optionally merges & pushes to Hub.
- Chat training (chat/train.py) uses similar flow but specialized dialogues to produce a conversational assistant.
- Inference (chat/generate.py) loads merged or base model from Hub → tokenizes input → generates sequences → decodes & returns responses.
- merge_peft_adapters.py takes base model + adapter checkpoint → merges weights → writes fused model → optionally pushes fused model back to Hub.

c. Architectural patterns & design principles
- Modular CLI scripts: each pipeline separated into its own namespace.
- Configuration-driven: altering behavior via YAML/JSON or CLI args without code changes.
- Reuse of core library functions (utils.py) to avoid duplication.
- Lazy dataset streaming for scalability.
- Adapter-based fine-tuning for parameter efficiency.

d. Key technologies & frameworks
- Python 3.x, Hugging Face Transformers, PEFT, bitsandbytes, datasets, accelerate, DeepSpeed
- Experiment tracking: wandb
- Deployment: Dockerized Text-Generation-Inference server (optional)

##### Elements
- From config → CLI scripts
- CLI scripts → core libs & deep learning frameworks
- Dataset → training loop → checkpoints → adapter merger → HF Hub → inference
- Inference → generates output to user
- Scripts (blue rectangles)
- Configs (green documents)
- External services (orange clouds)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration Layer
    subgraph "Configuration Layer"
        ConfigPY["config.py"]:::config
        ConfigYAML["config.yaml"]:::config
        DSConfig["deepspeed_z3_config_bf16.json"]:::config
    end

    %% CLI Scripts
    subgraph "CLI Scripts"
        TrainChat["chat/train.py"]:::script
        GenChat["chat/generate.py"]:::script
        Finetune["finetune/finetune.py"]:::script
        MergeAdapter["merge_peft_adapters.py"]:::script
    end

    %% Core Libraries
    subgraph "Core Libraries"
        ModelLoader["AutoModelForCausalLM & AutoTokenizer"]:::core
        PEFT["PEFT Adapters & BitsAndBytes"]:::core
        DatasetStream["Hugging Face Datasets"]:::core
        DeepSpeed["DeepSpeed (Z3, bf16)"]:::core
        Utils["utils.py"]:::core
    end

    %% External Services
    subgraph "External Services"
        HFHub["Hugging Face Hub"]:::external
        Wandb["wandb"]:::external
    end

    %% Data Stores
    subgraph "Data Stores"
        DataStore["Instruction Datasets"]:::datastore
        Checkpoints["./checkpoints"]:::datastore
    end

    %% Documentation
    subgraph "Documentation"
        ProjectReadme["README.md"]:::doc
        ChatReadme["chat/README.md"]:::doc
        ReqRoot["requirements.txt"]:::doc
        ReqChat["chat/requirements.txt"]:::doc
        License["LICENSE"]:::doc
        GitIgnore[".gitignore"]:::doc
    end

    %% Relationships
    ConfigPY -->|uses| TrainChat
    ConfigYAML -->|loads| TrainChat
    DSConfig -->|applies| TrainChat
    ConfigPY -->|uses| GenChat
    ConfigYAML -->|loads| GenChat
    ConfigPY -->|uses| Finetune
    ConfigPY -->|uses| MergeAdapter

    TrainChat -->|imports| Utils
    GenChat -->|imports| Utils
    Finetune -->|imports| Utils
    MergeAdapter -->|imports| Utils

    TrainChat -->|loads model| ModelLoader
    Finetune -->|loads model| ModelLoader
    GenChat -->|loads model| ModelLoader
    MergeAdapter -->|loads model| ModelLoader

    TrainChat -->|streams data| DatasetStream
    Finetune -->|streams data| DatasetStream

    TrainChat -->|configures| PEFT
    Finetune -->|configures| PEFT
    TrainChat -->|accelerates| DeepSpeed
    Finetune -->|accelerates| DeepSpeed

    DatasetStream -->|provides| DataStore
    TrainChat -->|writes checkpoints| Checkpoints
    Finetune -->|writes checkpoints| Checkpoints

    Checkpoints -->|adapter weights| MergeAdapter
    MergeAdapter -->|merged model| HFHub
    Finetune -->|push adapters| HFHub
    TrainChat -->|push adapters| HFHub

    GenChat -->|fetches model| HFHub
    GenChat -->|generates| userOutput["Generated Text"]

    TrainChat -->|logs metrics| Wandb
    Finetune -->|logs metrics| Wandb

    %% Documentation access
    ProjectReadme --> TrainChat
    ChatReadme --> TrainChat

    %% Click Events
    click ConfigPY "https://github.com/bigcode-project/starcoder/blob/main/chat/config.py"
    click ConfigYAML "https://github.com/bigcode-project/starcoder/blob/main/chat/config.yaml"
    click DSConfig "https://github.com/bigcode-project/starcoder/blob/main/chat/deepspeed_z3_config_bf16.json"
    click TrainChat "https://github.com/bigcode-project/starcoder/blob/main/chat/train.py"
    click GenChat "https://github.com/bigcode-project/starcoder/blob/main/chat/generate.py"
    click Finetune "https://github.com/bigcode-project/starcoder/blob/main/finetune/finetune.py"
    click MergeAdapter "https://github.com/bigcode-project/starcoder/blob/main/finetune/merge_peft_adapters.py"
    click Utils "https://github.com/bigcode-project/starcoder/blob/main/chat/utils.py"
    click ProjectReadme "https://github.com/bigcode-project/starcoder/blob/main/README.md"
    click ChatReadme "https://github.com/bigcode-project/starcoder/blob/main/chat/README.md"
    click ReqRoot "https://github.com/bigcode-project/starcoder/blob/main/requirements.txt"
    click ReqChat "https://github.com/bigcode-project/starcoder/blob/main/chat/requirements.txt"
    click License "https://github.com/bigcode-project/starcoder/tree/main/LICENSE"
    click GitIgnore "https://github.com/bigcode-project/starcoder/blob/main/.gitignore"

    %% Styles
    classDef script fill:#cce5ff,stroke:#004085,stroke-width:1px;
    classDef config fill:#d4edda,stroke:#155724,stroke-width:1px;
    classDef core fill:#e2e3e5,stroke:#383d41,stroke-width:1px;
    classDef external fill:#fff3cd,stroke:#856404,stroke-width:1px;
    classDef datastore fill:#d1ecf1,stroke:#0c5460,stroke-width:1px;
    classDef doc fill:#f8d7da,stroke:#721c24,stroke-width:1px;
    class TrainChat,GenChat,Finetune,MergeAdapter script
    class ConfigPY,ConfigYAML,DSConfig config
    class ModelLoader,PEFT,DatasetStream,DeepSpeed,Utils core
    class HFHub,Wandb external
    class DataStore,Checkpoints datastore
    class ProjectReadme,ChatReadme,ReqRoot,ReqChat,License,GitIgnore doc
```
