How haseebaslam95/mt-pkdot works
Project Type and Purpose
• This is a research/codebase project for Multi Target Privileged Knowledge Distillation using Optimal Transport (MT-PKDOT). It deals with training models – particularly a student network guided by a teacher – using both visual (face images) and physiological modalities. The project implements components that allow distributed data loading (with k-fold cross validation), model training, and evaluation.
• The README highlights sections for installation, dataset setup (with structured directories for face imagery), and running training pipelines. Multiple training scripts (e.g., mtpkdot.py, pkdot_kfold) indicate that the primary goal is optimizing the teacher-student distillation and alignment process.
Analyzing the File Structure
• Top-level files and directories include Python scripts (like mtpkdot.py, biovid_pkd.py), a models directory (containing various architectural definitions for attention models, transformation networks, etc.), dataset files, and utility modules (data_handler.py, mtpkdot_utils.py, physio_transforms.py).
• The models directory is key—it consists of different variants like Jcam_512, audguide_att, cam, cross-attention models (including dense_coattn, orig_crossattention_model), indicating multiple neural network architectures being experimented with.
• Auxiliary files such as sinkhorn.py and mmd_loss.py suggest that the system incorporates advanced loss functions (optimal transport and MMD loss) as part of the distillation process.
• Scripts like create_annotationstxt.py establish dataset structure by processing the folder hierarchy into annotation files for training.
a. Main Components:
• Data Ingestion & Preprocessing:
• Source Data: The dataset (folder “data”) and image folders structured according to classes.
• Annotation Creation: create_annotationstxt.py and k-fold annotation generation.
• Data Handler & Transformation: data_handler.py and physio_transforms.py manage reading data and preprocessing physiological signals.
• Dataset Loader: video_dataset_mm.py provides the data loaders that supply both visual and physiological modalities.
• Model Definitions:
• Models: The “models” directory holds various deep network architectures including attention-based models (e.g., cam.py, audguide_att.py, dense_coattn.py). This is where the teacher and student network architectures are defined.
• Transformation Modules: Files such as transformation_network.py, layer.py also play a role in adjusting the representations.
• Training Pipeline:
• Main Training Script: mtpkdot.py – the primary entry point for the training pipeline that runs student training and distillation.
• Distillation Techniques: The use of sophisticated loss functions (mmd_loss.py, sinkhorn.py) for aligning features across modalities; mtpkdot_utils.py for similarity matrices and visualization.
• Variations: Other training scripts (e.g., biovid_pkd.py, biovid_feat_mm_transformer_wtn_kfold.py) indicate perhaps alternative experimental setups.
• Evaluation & Validation:
• validate.py for running model evaluation on preprocessed validation sets.
b. Relationships and Interactions:
• Data Loader feeds prepared input (visual and physiological data) into the training pipeline.
• The Training Engine (mtpkdot.py, pkdot_kfold-based methods) utilizes the loaded data and passes it through the defined models. Loss modules like sinkhorn and mmd_loss are computed here.
• Model updates and optimization occur, and the outputs are then passed onto validation/evaluation modules.
• Optionally, teacher model inputs (pretrained and loaded from specified paths) provide guidance for the student network during distillation.
c. Architectural Patterns and Design Principles:
• Clear separation between Data Preparation, Model Definition, and Training/Evaluation.
• Modularity: Each component (data loading, transformation, model computation, loss evaluation) is implemented as separate scripts/functions, which improves maintainability and experiment reproducibility.
• Use of k-fold cross validation indicates a focus on robust evaluation.
• Incorporation of optimal transport (via sinkhorn) and MMD losses shows advanced loss engineering in the training loop.
d. Technologies, Frameworks, and Libraries:
• PyTorch (specifically version 1.10.0 as noted) for deep learning model construction and training.
• Common Python libraries as referenced by the requirements.txt.
• External services might not be directly involved, but the data annotation and preparation suggest integration with a standardized file/folder dataset structure.
Final Inclusions
• Design patterns like modular design or use of k-fold cross validation.
• Mention the main frameworks used (i.e., PyTorch) near the training components.