# instructlab/training architecture

> Architecture diagram of the GitHub repository instructlab/training, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/instructlab/training
- Repository: https://github.com/instructlab/training

## Overview

##### Project type and purpose
- This is an open-source Python library (“InstructLab Training”) for instruction-tuning large language models (LLMs) on messages-format data.
- Purpose: provide a simple, pythonic interface (`run_training`) and CLI for distributed fine-tuning (via DeepSpeed or PyTorch FSDP), data pre-processing, tokenization, checkpointing, logging, and support for “reasoning_content” traces.

##### File structure
- Top-level:
- src/instructlab/training: core modules (accelerator, model, data_process, multipack_sampler, token_dataset, config, utils, logger, async_logger, main_ds entrypoint, chat_templates).
- scripts/: helper scripts for data tests and graphs.
- examples/: Jupyter notebooks and Python demos showing usage.
- sample-data/: sample JSONL training data.
- tests/: unit and smoke tests.
- docs/: CI, logging, reasoning content guides.
- CI and GitHub Actions workflows under .github/workflows.
- config files: pyproject.toml, requirements*.txt, Makefile, tox.ini.
- Patterns:
- Separation between data processing (src/.../data_process.py) and training orchestration (main_ds.py → run_training).
- Modular chat template plugin directory (chat_templates).
- Pluggable distributed backends via TorchrunArgs and TrainingArgs.

##### README for additional insights
- Describes two main entrypoints:
a) `run_training(torchrun_args, training_args)`
b) `data_process` for separate pre-processing.
- Highlights key subsystems: TrainingArgs, DeepSpeedOptions, FSDPOptions, LoraOptions, TorchrunArgs.
- Emphasizes support for reasoning_content, configurable optimizers, checkpointing, logging.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Interface / API
    subgraph "User Interface / API"
        UI["Python API\nrun_training()"]:::python
        CLI["Examples & CLI Scripts"]:::python
    end

    %% Configuration Layer
    subgraph "Configuration Layer"
        CFG["TrainingArgs,\nTorchrunArgs,\nDeepSpeedOptions,\nFSDPOptions,\nLoraOptions"]:::python
        TD_TYPES["Type Definitions"]:::python
    end

    %% Launcher
    subgraph "Launcher"
        TR["torchrun Launcher"]:::external
    end

    %% Data Processing Path
    subgraph "Data Processing"
        DP["Data Processing\n(JSONL → tokens)"]:::python
        ILAB["ILab to SDG Converter"]:::python
    end

    %% Data Layer
    subgraph "Data Layer"
        TOKEN["TokenDataset"]:::python
        SAMPLER["multipack_sampler"]:::python
        TOKEN_UTILS["Tokenizer Utilities"]:::python
    end

    %% Training Layer
    subgraph "Training Engine"
        MODEL["Model & Templates Loader"]:::python
        ACCEL["Accelerator Abstraction\n(DeepSpeed/FSDP)"]:::python
        PLUGINS["Chat Templates Plugin"]:::python
    end

    %% Logging & Monitoring
    subgraph "Logging & Monitoring"
        LOG["Logger"]:::python
        ASYNC["Async Logger"]:::python
    end

    %% Checkpoint & Output
    subgraph "Checkpoint & Output"
        CP["Checkpointing & I/O"]:::python
    end

    %% Infrastructure
    subgraph "Infrastructure"
        HW["GPU Nodes,\nCUDA, NCCL"]:::infra
        STORAGE["Shared Storage"]:::infra
    end

    %% Connections
    UI --> CFG
    CLI --> CFG
    CFG --> TR
    TR -->|"launches"| DP
    TR -->|"launches"| TOKEN
    DP --> ILAB
    ILAB --> TOKEN
    TOKEN_UTILS --> TOKEN
    TOKEN --> SAMPLER
    SAMPLER --> ACCEL
    ACCEL --> MODEL
    PLUGINS --> MODEL
    ACCEL --> CP
    ACCEL --> LOG
    LOG --> ASYNC
    CP --> STORAGE
    ACCEL --> HW
    HW --> ACCEL

    %% Click Events
    click UI "https://github.com/instructlab/training/blob/main/src/instructlab/training/main_ds.py"
    click CLI "https://github.com/instructlab/training/tree/main/examples/"
    click CFG "https://github.com/instructlab/training/blob/main/src/instructlab/training/config.py"
    click TD_TYPES "https://github.com/instructlab/training/blob/main/src/instructlab/training/type_definitions.py"
    click TR "https://github.com/instructlab/training/blob/main/src/instructlab/training/main_ds.py"
    click DP "https://github.com/instructlab/training/blob/main/src/instructlab/training/data_process.py"
    click ILAB "https://github.com/instructlab/training/blob/main/src/instructlab/training/ilab_to_sdg.py"
    click TOKEN "https://github.com/instructlab/training/blob/main/src/instructlab/training/token_dataset.py"
    click SAMPLER "https://github.com/instructlab/training/blob/main/src/instructlab/training/multipack_sampler.py"
    click TOKEN_UTILS "https://github.com/instructlab/training/blob/main/src/instructlab/training/tokenizer_utils.py"
    click MODEL "https://github.com/instructlab/training/blob/main/src/instructlab/training/model.py"
    click ACCEL "https://github.com/instructlab/training/blob/main/src/instructlab/training/accelerator.py"
    click PLUGINS "https://github.com/instructlab/training/tree/main/src/instructlab/training/chat_templates/"
    click LOG "https://github.com/instructlab/training/blob/main/src/instructlab/training/logger.py"
    click ASYNC "https://github.com/instructlab/training/blob/main/src/instructlab/training/async_logger.py"
    click CP "https://github.com/instructlab/training/blob/main/src/instructlab/training/utils.py"
    click CLI "https://github.com/instructlab/training/tree/main/examples/"

    %% Styles
    classDef python fill:#D0E8FF,stroke:#0366d6,color:#034694
    classDef external fill:#CFFFCF,stroke:#119955,color:#096628
    classDef infra fill:#F0F0F0,stroke:#888888,color:#333333
```
