How agokrani/distillkitplus works
Main Components
• Core Distillation Modules: These are housed inside the “components” directory and include modules for configuration (config.py), handling datasets (dataset.py), formatting (formatters.py), model definitions (models.py), and the training process (trainer.py).
• Configuration Layer: The “config” directory contains various config files:
– Default configuration for projects (default_config.json)
– Specialized configurations for distributed training using Accelerate (YAML files under accelerate_configs) and DeepSpeed (JSON files under deepspeed_configs).
• Execution Scripts:
– The “scripts” directory is subdivided into two regions: “local” and “modal”. “Local” scripts (e.g., generate_logits.py and distill_logits.py) enable running the toolkit from a typical command-line environment. “Modal” scripts provide an integration for running on Modal, internally handling distribute settings and execution configurations based on the config file.
• Setup and Dependencies: Files like requirements.txt, setup.py, and the README indicate how to install and initialize the toolkit, which frames the project as a Python-based distillation tool.
Relationships and Interactions
• The execution scripts (in both “local” and “modal”) serve as entry points. They load configuration settings from the JSON and YAML files (found in the “config” directory) to determine how to set up training and distillation sessions.
• These scripts interact with the core functionalities implemented in the “components” folder. For instance, the scripts invoke functions for generating teacher logits (using models and dataset handling) and then kick off the distillation process using the trainer module.
• The configuration files dictate runtime behaviors that may include distributed training parameters (via Accelerate and DeepSpeed integrations) and optional Modal execution.
Architectural Patterns and Design Principles
• Modular Architecture: Separation of concerns is evident by dividing the project into clearly defined modules (components for core logic, config for settings, and scripts for execution).
• Configurable Execution: By externalizing key parameters into JSON and YAML config files, the system supports dynamic switching between local execution, distributed training, and Modal integration.
• Extensibility: The toolkit supports multiple execution environments (local vs. modal) and advanced functionalities (logit distillation, LoRA, quantization) that are extensible with minimal changes to the codebase.
Relevant Technologies and Libraries
• Python is the main programming language.
• Accelerate and DeepSpeed are integrated for efficient distributed training.
• Modal is utilized as an optional platform to run scripts in an alternative cloud or remote environment.
• Additional libraries likely include those for deep learning model manipulation, dataset processing, and optimization.