How browniannotion/bitdistiller works
Main Components
• Environment & Setup: This includes the initial setup with the shell scripts (setup.sh) and configuration files (e.g., train/config/*.json).
• Data Generation & Preparation: Located in data/generation and data/datasets. This part of the system is responsible for generating teacher data (e.g., via generate.sh and mix_data.py), and managing datasets (mix_wiki_alpaca_8000.json, etc.).
• Quantization & Clipping: Found under the quantization directory and some pieces inside data/generation. This component invokes tools (like autoclip.py) for clipping/quantising the teacher model to generate initial quantized weights.
• Training Pipeline: Under the train directory (with train.py, train.sh, and trainer configuration files). This module orchestrates the training workflow by applying distillation and possibly fine-tuning on the quantized weights.
• Inference & Model Operations: Under inference. This includes core engine implementations – with CUDA kernels for attention, layer normalization, position embeddings, and quantization kernels (mix of C++, CUDA, and Python modules) that help run the model efficiently.
• Evaluation & Testing: Scripts and tests contained in test (especially within test/general) that run various benchmarks (PPL, QA tasks, MMLU style evaluations, etc.) and also generate and upload metrics.
• Model Upload & Metrics Visualization: The upload_model.py and upload_metrics.py scripts in the root and test directories handle publishing the trained models to Hugging Face, while the visualizations directory (with plotting scripts) supports analysis of weight distributions and training dynamics.
Out Relationships and Data Flow
• The Setup phase is the entry point, establishing the environment.
• From there, the system flows to Pre-Training:
– Data Generation creates teacher outputs based on a chosen dataset.
– Clipping/Quantization converts a teacher model (e.g., TinyLlama_v1.1) into a quantized version using tools like autoclip.py.
• The Training pipeline takes the generated teacher data and initial quantized weights as input parameters, running training scripts that update the model through self-distillation.
• Once a model checkpoint is produced, the Inference component leverages specialized CUDA kernels (attention, layernorm, and quantization) to serve or run the model.
• Concurrently or subsequently, the Evaluation scripts use the trained model to calculate metrics and benchmarking results.
Incorporate Important Architectural Patterns
• Modular Design: Notice how separate directories (data, quantization, train, inference, test) help encapsulate distinct concerns.
• Pipeline Workflow: A clear sequential pipeline from environment setup –> pre-training (data generation & clipping) –> training –> inference –> evaluation/upload.
• Hardware-Accelerated Computing: The system design makes heavy use of GPU-based acceleration with custom CUDA kernels for inference.
• Extensibility: The project has plug-in like structure for different tasks (via tests under test/general, support for multiple datasets, etc.) and model uploads.
Relevant Technologies, Frameworks, and Libraries
• CUDA: Underpins custom kernels found in inference/kernels (attention, layernorm, position embedding, and quantization).
• Python: Drives data generation, training, evaluation, and orchestration scripts.
• Shell Scripting (bash): Used in setup, training, and evaluation routines.
• TensorBoard: For monitoring the training progress.
• Hugging Face Integration: For uploading models and metrics.
• Possibly use of quantization frameworks, custom modules (qlinear, qmodule, etc.) and environment management with a virtual environment (BitDistillerVenv).