# haseebaslam95/mt-pkdot architecture

> Architecture diagram of the GitHub repository haseebaslam95/mt-pkdot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/haseebaslam95/mt-pkdot
- Repository: https://github.com/haseebaslam95/mt-pkdot

## Overview

##### Project Type and Purpose
- This is a research/codebase project for Multi Target Privileged Knowledge Distillation using Optimal Transport (MT-PKDOT). It deals with training models – particularly a student network guided by a teacher – using both visual (face images) and physiological modalities. The project implements components that allow distributed data loading (with k-fold cross validation), model training, and evaluation.
- The README highlights sections for installation, dataset setup (with structured directories for face imagery), and running training pipelines. Multiple training scripts (e.g., mtpkdot.py, pkdot_kfold) indicate that the primary goal is optimizing the teacher-student distillation and alignment process.

##### Analyzing the File Structure
- Top-level files and directories include Python scripts (like mtpkdot.py, biovid_pkd.py), a models directory (containing various architectural definitions for attention models, transformation networks, etc.), dataset files, and utility modules (data_handler.py, mtpkdot_utils.py, physio_transforms.py).
- The models directory is key—it consists of different variants like Jcam_512, audguide_att, cam, cross-attention models (including dense_coattn, orig_crossattention_model), indicating multiple neural network architectures being experimented with.
- Auxiliary files such as sinkhorn.py and mmd_loss.py suggest that the system incorporates advanced loss functions (optimal transport and MMD loss) as part of the distillation process.
- Scripts like create_annotationstxt.py establish dataset structure by processing the folder hierarchy into annotation files for training.

a. Main Components:
- Data Ingestion & Preprocessing:
- Source Data: The dataset (folder “data”) and image folders structured according to classes.
- Annotation Creation: create_annotationstxt.py and k-fold annotation generation.
- Data Handler & Transformation: data_handler.py and physio_transforms.py manage reading data and preprocessing physiological signals.
- Dataset Loader: video_dataset_mm.py provides the data loaders that supply both visual and physiological modalities.

- Model Definitions:
- Models: The “models” directory holds various deep network architectures including attention-based models (e.g., cam.py, audguide_att.py, dense_coattn.py). This is where the teacher and student network architectures are defined.
- Transformation Modules: Files such as transformation_network.py, layer.py also play a role in adjusting the representations.

- Training Pipeline:
- Main Training Script: mtpkdot.py – the primary entry point for the training pipeline that runs student training and distillation.
- Distillation Techniques: The use of sophisticated loss functions (mmd_loss.py, sinkhorn.py) for aligning features across modalities; mtpkdot_utils.py for similarity matrices and visualization.
- Variations: Other training scripts (e.g., biovid_pkd.py, biovid_feat_mm_transformer_wtn_kfold.py) indicate perhaps alternative experimental setups.

- Evaluation & Validation:
- validate.py for running model evaluation on preprocessed validation sets.

b. Relationships and Interactions:
- Data Loader feeds prepared input (visual and physiological data) into the training pipeline.
- The Training Engine (mtpkdot.py, pkdot_kfold-based methods) utilizes the loaded data and passes it through the defined models. Loss modules like sinkhorn and mmd_loss are computed here.
- Model updates and optimization occur, and the outputs are then passed onto validation/evaluation modules.
- Optionally, teacher model inputs (pretrained and loaded from specified paths) provide guidance for the student network during distillation.

c. Architectural Patterns and Design Principles:
- Clear separation between Data Preparation, Model Definition, and Training/Evaluation.
- Modularity: Each component (data loading, transformation, model computation, loss evaluation) is implemented as separate scripts/functions, which improves maintainability and experiment reproducibility.
- Use of k-fold cross validation indicates a focus on robust evaluation.
- Incorporation of optimal transport (via sinkhorn) and MMD losses shows advanced loss engineering in the training loop.

d. Technologies, Frameworks, and Libraries:
- PyTorch (specifically version 1.10.0 as noted) for deep learning model construction and training.
- Common Python libraries as referenced by the requirements.txt.
- External services might not be directly involved, but the data annotation and preparation suggest integration with a standardized file/folder dataset structure.

##### Final Inclusions
- Design patterns like modular design or use of k-fold cross validation.
- Mention the main frameworks used (i.e., PyTorch) near the training components.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        D1["data (Raw Data Storage)"]:::data
        D2["create_annotationstxt.py (Annotation Creation)"]:::data
        D3["data_handler.py (Data Handling)"]:::data
        D4["physio_transforms.py (Physiological Transformation)"]:::data
    end

    D1 -->|"raw_images"| D2
    D2 -->|"annotations"| D3
    D3 -->|"preprocess"| D4

    %% Data Loader
    subgraph "Data Loader"
        DL["video_dataset_mm.py (Data Loader Module)"]:::data
    end

    %% Model Definitions
    subgraph "Model Definitions"
        M1["models/cam.py"]:::model
        M2["models/audguide_att.py"]:::model
        M3["models/dense_coattn.py"]:::model
        M4["models/transformation_network.py"]:::model
        M5["models/layer.py"]:::model
    end

    %% Training Pipeline
    subgraph "Training Pipeline"
        T1["mtpkdot.py (Main Training Script)"]:::training
        T2["biovid_pkd.py (Alternative Training Variant)"]:::training
        T3["biovid_feat_mm_transformer_wtn_kfold.py (Alt Training Variant)"]:::training
        T4["mtpkdot_utils.py (Utilities & Distillation)"]:::training
    end

    %% Loss Functions & Optimization
    subgraph "Loss Functions & Optimization"
        L1["sinkhorn.py (Optimal Transport Loss)"]:::training
        L2["mmd_loss.py (MMD Loss)"]:::training
    end

    %% Evaluation & Validation
    subgraph "Evaluation & Validation"
        E1["validate.py (Evaluation Script)"]:::evaluation
    end

    %% Connections between major components
    D4 -->|"preprocessed_data"| DL
    DL -->|"loaded_data"| T1
    T1 -->|"model_forward"| M1
    T1 -->|"model_forward"| M2
    T1 -->|"model_forward"| M3
    T1 -->|"model_forward"| M4
    T1 -->|"model_forward"| M5
    T1 -->|"compute_losses"| L1
    T1 -->|"compute_losses"| L2
    T1 -->|"evaluation"| E1

    %% Optional links for alternative training variants
    DL --> T2
    DL --> T3

    %% Click Events
    click D1 "https://github.com/haseebaslam95/mt-pkdot/tree/master/data"
    click D2 "https://github.com/haseebaslam95/mt-pkdot/blob/master/create_annotationstxt.py"
    click D3 "https://github.com/haseebaslam95/mt-pkdot/blob/master/data_handler.py"
    click D4 "https://github.com/haseebaslam95/mt-pkdot/blob/master/physio_transforms.py"
    click DL "https://github.com/haseebaslam95/mt-pkdot/blob/master/video_dataset_mm.py"
    click M1 "https://github.com/haseebaslam95/mt-pkdot/blob/master/models/cam.py"
    click M2 "https://github.com/haseebaslam95/mt-pkdot/blob/master/models/audguide_att.py"
    click M3 "https://github.com/haseebaslam95/mt-pkdot/blob/master/models/dense_coattn.py"
    click M4 "https://github.com/haseebaslam95/mt-pkdot/blob/master/models/transformation_network.py"
    click M5 "https://github.com/haseebaslam95/mt-pkdot/blob/master/models/layer.py"
    click T1 "https://github.com/haseebaslam95/mt-pkdot/blob/master/mtpkdot.py"
    click T2 "https://github.com/haseebaslam95/mt-pkdot/blob/master/biovid_pkd.py"
    click T3 "https://github.com/haseebaslam95/mt-pkdot/blob/master/biovid_feat_mm_transformer_wtn_kfold.py"
    click T4 "https://github.com/haseebaslam95/mt-pkdot/blob/master/mtpkdot_utils.py"
    click L1 "https://github.com/haseebaslam95/mt-pkdot/blob/master/sinkhorn.py"
    click L2 "https://github.com/haseebaslam95/mt-pkdot/blob/master/mmd_loss.py"
    click E1 "https://github.com/haseebaslam95/mt-pkdot/blob/master/validate.py"

    %% Styles
    classDef data fill:#b3cde0,stroke:#333,stroke-width:2px;
    classDef model fill:#ccebc5,stroke:#333,stroke-width:2px;
    classDef training fill:#fddb94,stroke:#333,stroke-width:2px;
    classDef evaluation fill:#d9d2e9,stroke:#333,stroke-width:2px;
```
