How ghassen-fatnassi/actia_research_internship works
Project Type & Purpose
• This is a machine learning research project that develops and tests methods for knowledge distillation in semantic segmentation.
• It integrates components for model training, loss function exploration, distributed training, and experiment monitoring using libraries like Accelerate and Weights & Biases.
• The README details experiments with teacher–student training paradigms and methods such as online distillation, representational KD, and response KD.
Analyzing the File Structure
• The top-level “workstation” directory contains a “src” folder that is clearly the heart of the system.
• Within “src” there are several key modules:
• Core training and inference scripts: train.py, teacher_training.py, response_distillation.py.
• Dataset handling: dataset.py.
• Loss and KPI modules: loss.py, benchmark.py.
• Utility functions: utils.py.
• A “config” folder holds YAML files that configure different experiments (e.g., SegFormer.yaml, Unet.yaml, config.yaml).
• The “main” directory under “src/main/Unet” contains submodules representing different distillation strategies (onlineKD, representational_KD, response_KD). This indicates a modular design where various distillation types are encapsulated separately.
• Additionally, the presence of GitHub workflow configs and requirements.txt shows standard software project practices.
a. Main Components:
• Data Layer: Represented by the dataset.py file which handles dataset loading and preparation.
• Configuration Management: Illustrate the config files (SegFormer.yaml, Unet.yaml, config.yaml) that parameterize experiments.
• Model Architecture:
• Base UNet model as the central segmentation architecture.
• Teacher model training (teacher_training.py) and Student model training (train.py) modules.
• Distillation Modules:
• Online Distillation: Modules in the onlineKD folder.
• Representational Distillation: Represented by the content under representational_KD.
• Response Distillation: Indicated by files like response_distillation.py and folders such as response_KD.
• Loss Functions: Encapsulated in loss.py, supporting customized losses including cross-entropy and dark knowledge-enhanced losses.
• Experiment Management & Monitoring:
• Distributed training support using Accelerate (as mentioned in README).
• Logging/Monitoring integrations (Weights & Biases).
• Benchmarking tools (benchmark.py) for performance comparisons.
• Utility Functions: Located in utils.py for shared operations across modules.
b. Relationships & Interactions:
• The configuration system feeds into training scripts (train.py, teacher_training.py) guiding model setup.
• The UNet-based architecture forms the core, with branching submodules applying different distillation paradigms.
• Loss functions interact with both teacher and student training modules.
• Experiment monitoring and evaluation (benchmark.py, logging integrations) occur alongside training.
c. Important Architectural Patterns:
• Modularity: The separation of distinct distillation strategies (online, representational, response) into individual modules.
• Configuration-Driven: Use of YAMLs for parameterizing experiments.
• Pipeline Structure: The project exhibits a clear ML pipeline starting with data preparation, model training (teacher and student), loss optimization, and finally distributed training and evaluation.
d. Technologies and Frameworks:
• PyTorch as the primary deep learning framework.
• Distributed training via the Accelerate library.
• Experiment monitoring via Weights & Biases.
• Standard Python components for utilities, configuration parsing, and workflow integration (GitHub actions for CodeQL scanning).