# browniannotion/bitdistiller architecture

> Architecture diagram of the GitHub repository browniannotion/bitdistiller, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/browniannotion/bitdistiller
- Repository: https://github.com/browniannotion/bitdistiller

## Overview

##### Main Components
- Environment & Setup: This includes the initial setup with the shell scripts (setup.sh) and configuration files (e.g., train/config/*.json).
- Data Generation & Preparation: Located in data/generation and data/datasets. This part of the system is responsible for generating teacher data (e.g., via generate.sh and mix_data.py), and managing datasets (mix_wiki_alpaca_8000.json, etc.).
- Quantization & Clipping: Found under the quantization directory and some pieces inside data/generation. This component invokes tools (like autoclip.py) for clipping/quantising the teacher model to generate initial quantized weights.
- Training Pipeline: Under the train directory (with train.py, train.sh, and trainer configuration files). This module orchestrates the training workflow by applying distillation and possibly fine-tuning on the quantized weights.
- Inference & Model Operations: Under inference. This includes core engine implementations – with CUDA kernels for attention, layer normalization, position embeddings, and quantization kernels (mix of C++, CUDA, and Python modules) that help run the model efficiently.
- Evaluation & Testing: Scripts and tests contained in test (especially within test/general) that run various benchmarks (PPL, QA tasks, MMLU style evaluations, etc.) and also generate and upload metrics.
- Model Upload & Metrics Visualization: The upload_model.py and upload_metrics.py scripts in the root and test directories handle publishing the trained models to Hugging Face, while the visualizations directory (with plotting scripts) supports analysis of weight distributions and training dynamics.

##### Out Relationships and Data Flow
- The Setup phase is the entry point, establishing the environment.
- From there, the system flows to Pre-Training:
– Data Generation creates teacher outputs based on a chosen dataset.
– Clipping/Quantization converts a teacher model (e.g., TinyLlama_v1.1) into a quantized version using tools like autoclip.py.
- The Training pipeline takes the generated teacher data and initial quantized weights as input parameters, running training scripts that update the model through self-distillation.
- Once a model checkpoint is produced, the Inference component leverages specialized CUDA kernels (attention, layernorm, and quantization) to serve or run the model.
- Concurrently or subsequently, the Evaluation scripts use the trained model to calculate metrics and benchmarking results.

##### Incorporate Important Architectural Patterns
- Modular Design: Notice how separate directories (data, quantization, train, inference, test) help encapsulate distinct concerns.
- Pipeline Workflow: A clear sequential pipeline from environment setup –> pre-training (data generation & clipping) –> training –> inference –> evaluation/upload.
- Hardware-Accelerated Computing: The system design makes heavy use of GPU-based acceleration with custom CUDA kernels for inference.
- Extensibility: The project has plug-in like structure for different tasks (via tests under test/general, support for multiple datasets, etc.) and model uploads.

##### Relevant Technologies, Frameworks, and Libraries
- CUDA: Underpins custom kernels found in inference/kernels (attention, layernorm, position embedding, and quantization).
- Python: Drives data generation, training, evaluation, and orchestration scripts.
- Shell Scripting (bash): Used in setup, training, and evaluation routines.
- TensorBoard: For monitoring the training progress.
- Hugging Face Integration: For uploading models and metrics.
- Possibly use of quantization frameworks, custom modules (qlinear, qmodule, etc.) and environment management with a virtual environment (BitDistillerVenv).

## Diagram (Mermaid)

```mermaid
graph TD
    %% Environment & Setup
    subgraph "Environment & Setup"
        E1("setup.sh"):::shell
        E2("Config: train/config/zero.json"):::config
    end

    %% Data Generation & Preparation
    subgraph "Data Generation & Preparation"
        D1("Gen: data/generation/generate.sh"):::shell
        D2("Gen: data/generation/mix_data.py"):::python
        D3("Dataset: data/datasets/tinyllama_v1.1"):::dataset
    end

    %% Quantization & Clipping
    subgraph "Quantization & Clipping"
        Q1("Tool: quantization/autoclip.py"):::python
        Q2("Script: quantization/clip.sh"):::shell
        Q3("Prep: quantization/pre_quant.py"):::python
        Q4("Module: quantization/qlinear.py"):::python
    end

    %% Training Pipeline
    subgraph "Training Pipeline"
        T1("Script: train/train.py"):::python
        T2("Shell: train/train.sh"):::shell
        T3("Trainer: train/mytrainer.py"):::python
    end

    %% Inference & Model Operations
    subgraph "Inference & Model Operations"
        I1("Demo: inference/demo.py"):::python
        I2("Dump: inference/dump_quant.py"):::python
        I3("Kernels: inference/kernels"):::cuda
        I4("Ops: inference/models"):::python
    end

    %% Evaluation & Testing
    subgraph "Evaluation & Testing"
        EV1("Metrics: test/general/generate_metrics.sh"):::shell
        EV2("Eval: test/general/llm_eval.py"):::python
        EV3("Suite: test/general/lm_eval"):::test
    end

    %% Model Upload & Metrics Visualization
    subgraph "Model Upload & Metrics Visualization"
        U1("Upload: upload_model.py"):::python
        U2("Plots: visualizations/plots"):::python
        U3("Weights: visualizations/weight_distribution.py"):::python
        U4("UploadMetrics: test/general/upload_metrics.py"):::python
    end

    %% Data Flow
    E1 -->|"setupDone"| D1
    D3 -->|"dataForQuant"| Q1
    Q4 -->|"quantizedWeights"| T1
    T3 -->|"trainedModel"| I1
    T3 -->|"trainedModel"| EV1
    EV3 -->|"evaluationResults"| U1

    %% Click Events
    click E1 "https://github.com/browniannotion/bitdistiller/blob/main/setup.sh"
    click E2 "https://github.com/browniannotion/bitdistiller/blob/main/train/config/zero.json"
    click D1 "https://github.com/browniannotion/bitdistiller/blob/main/data/generation/generate.sh"
    click D2 "https://github.com/browniannotion/bitdistiller/blob/main/data/generation/mix_data.py"
    click D3 "https://github.com/browniannotion/bitdistiller/blob/main/data/datasets/tinyllama_v1.1"
    click Q1 "https://github.com/browniannotion/bitdistiller/blob/main/quantization/autoclip.py"
    click Q2 "https://github.com/browniannotion/bitdistiller/blob/main/quantization/clip.sh"
    click Q3 "https://github.com/browniannotion/bitdistiller/blob/main/quantization/pre_quant.py"
    click Q4 "https://github.com/browniannotion/bitdistiller/blob/main/quantization/qlinear.py"
    click T1 "https://github.com/browniannotion/bitdistiller/blob/main/train/train.py"
    click T2 "https://github.com/browniannotion/bitdistiller/blob/main/train/train.sh"
    click T3 "https://github.com/browniannotion/bitdistiller/blob/main/train/mytrainer.py"
    click I1 "https://github.com/browniannotion/bitdistiller/blob/main/inference/demo.py"
    click I2 "https://github.com/browniannotion/bitdistiller/blob/main/inference/dump_quant.py"
    click I3 "https://github.com/browniannotion/bitdistiller/tree/main/inference/kernels"
    click I4 "https://github.com/browniannotion/bitdistiller/tree/main/inference/models"
    click EV1 "https://github.com/browniannotion/bitdistiller/blob/main/test/general/generate_metrics.sh"
    click EV2 "https://github.com/browniannotion/bitdistiller/blob/main/test/general/llm_eval.py"
    click EV3 "https://github.com/browniannotion/bitdistiller/tree/main/test/general/lm_eval"
    click U1 "https://github.com/browniannotion/bitdistiller/blob/main/upload_model.py"
    click U2 "https://github.com/browniannotion/bitdistiller/tree/main/visualizations/plots"
    click U3 "https://github.com/browniannotion/bitdistiller/blob/main/visualizations/weight_distribution.py"
    click U4 "https://github.com/browniannotion/bitdistiller/blob/main/test/general/upload_metrics.py"

    %% Styles
    classDef shell fill:#f4cccc, stroke:#cc0000, stroke-width:2px;
    classDef python fill:#c9daf8, stroke:#1155cc, stroke-width:2px;
    classDef config fill:#ead1dc, stroke:#b8549f, stroke-width:2px;
    classDef dataset fill:#d9ead3, stroke:#6aa84f, stroke-width:2px;
    classDef cuda fill:#fff2cc, stroke:#bf9000, stroke-width:2px;
    classDef test fill:#d0e0e3, stroke:#0b5394, stroke-width:2px;
```
