# leo2105/jointmultiunimodalattention architecture

> Architecture diagram of the GitHub repository leo2105/jointmultiunimodalattention, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/leo2105/jointmultiunimodalattention
- Repository: https://github.com/leo2105/jointmultiunimodalattention

## Overview

##### Project Type & Purpose
- This is a machine learning project utilizing deep learning (PyTorch) to perform attention-based fusion on image features. Its primary purpose is to jointly leverage multiple unimodal features using a fusion mechanism. The project supports different training/testing protocols (with/without LOSO, with/without PCA) as evident from the multiple main scripts.
- The Dockerfile (Dockerfile_pytorch) indicates an environment setup tailored for PyTorch, implying containerized deployment and reproducibility.

##### Analyzing the File Structure
- Top-level files and scripts (main.py, main_noloso.py, main_nopca.py) serve as entry points for different training and evaluation protocols.
- Utility or helper scripts like cam.py / cam_nopca.py, training.py / training_nopca.py, testing.py / testing_nopca.py, and validation.py / validation_nopca.py hint at a modular design where the process of building, training, testing, and validating models is split into smaller reusable components.
- The Dockerfile shows that the project is deployed in a container optimized for PyTorch, which is central to model training/inference.
- The presence of a pre-trained model file (model_cam.pth) indicates that a saved model is also part of the system.

##### Key Components
a. Environment & Configuration:
- Docker Container: Highlight the usage of Docker and the Dockerfile_pytorch to standardize the runtime environment.
- Environment Variables: Collect and configure parameters (pca_size, epochs, nro_rep, kind_rep, dataset_target, folder_path_rep) that tailor each training run.

b. Data Management:
- Dataset Loader: Represent the module responsible for reading and processing the datasets (as described in the README, e.g., the folder structure for JAFFE and CK).
- Preprocessing Module: A node indicating the PCA module that is optionally applied (evidenced by the differences in main scripts—using PCA or a fully-connected layer instead).

c. Training Pipeline:
- Model Architecture: A block representing the attention fusion network (unimodal attention with fusion) – possibly described by cam.py or its variants. This also includes storage and loading of pre-trained weights (model_cam.pth).

d. Validation & Testing:
- Validation Module: A node showing how validation is performed (validation.py / validation_nopca.py) to track performance during training.
- Testing Pipeline: A node showing testing routines (testing.py / testing_nopca.py) that conclude the experimental protocol.

e. Experiment Variants:
- Different Protocol Modes: Clearly mark the three main workflows:
- LOSO protocol with PCA (main.py)
- Without LOSO and with PCA (main_noloso.py)
- Without LOSO and without PCA (main_nopca.py), which uses a FC layer alternative.

f. Data Flow and Interactions:
- That model parameters (saved in model_cam.pth) are used in the inference pipelines (cam.py or cam_nopca.py) for performance evaluation.

##### Architectural Patterns / Design Principles
- Modularity: The split of functions into different scripts (training, testing, validation) suggests a modular design.
- Configurability: Use of environment variables to set parameters indicates an emphasis on configurable execution.
- Containerization: The Dockerfile establishes a reproducible and portable environment.

- Components:
- Docker Container as the encapsulating runtime environment.
- A Config/Env Variables component feeding into all other modules.
- Data Input Module (Dataset Loader + Preprocessing/PCA).
- Model & Training Module (attention fusion model, training routines).
- Validation Module.
- Testing/Inference Module.
- A branch for experiment protocol differentiation (three main entry points).
- Relationships:
- Data flow from the Dataset Loader to the Preprocessing Module, then into the Training Module.
- From the Training Module, branch out to the Validation (feedback loop) and then the Testing Module.
- Visual Techniques:
- Annotations specifying key environment variables and the role of PCA.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Execution Environment
    subgraph "Execution Environment"
        DC["Docker Container"]:::env
        ENV["Config/Environment Variables"]:::env
    end
    click DC "https://github.com/leo2105/jointmultiunimodalattention/tree/main/Dockerfile_pytorch"
    
    %% Experiment Protocols
    subgraph "Experiment Protocols"
        LOSO["LOSO protocol with PCA (main.py)"]:::protocol
        NoLOSO["Without LOSO with PCA (main_noloso.py)"]:::protocol
        NoPCA["Without LOSO without PCA (main_nopca.py)"]:::protocol
    end
    click LOSO "https://github.com/leo2105/jointmultiunimodalattention/blob/main/main.py"
    click NoLOSO "https://github.com/leo2105/jointmultiunimodalattention/blob/main/main_noloso.py"
    click NoPCA "https://github.com/leo2105/jointmultiunimodalattention/blob/main/main_nopca.py"
    
    %% Data Management
    subgraph "Data Management"
        DL["Dataset Loader"]:::data
        PP["Preprocessing Module (PCA)"]:::data
    end
    
    %% Training Pipeline
    subgraph "Training Pipeline"
        TP_PCA["Training (with PCA)"]:::training
        TP_noPCA["Training (without PCA)"]:::training
    end
    click TP_PCA "https://github.com/leo2105/jointmultiunimodalattention/blob/main/training.py"
    click TP_noPCA "https://github.com/leo2105/jointmultiunimodalattention/blob/main/training_nopca.py"
    
    %% Model Architecture & Inference
    subgraph "Model Architecture & Inference"
        MA_PCA["Attention Fusion Network (with PCA)"]:::model
        PT["Pre-trained Weights"]:::model
        MA_noPCA["Attention Fusion Network (without PCA)"]:::model
    end
    click MA_PCA "https://github.com/leo2105/jointmultiunimodalattention/blob/main/cam.py"
    click PT "https://github.com/leo2105/jointmultiunimodalattention/blob/main/model_cam.pth"
    click MA_noPCA "https://github.com/leo2105/jointmultiunimodalattention/blob/main/cam_nopca.py"
    
    %% Validation Module
    subgraph "Validation Module"
        VAL_PCA["Validation (with PCA)"]:::validation
        VAL_noPCA["Validation (without PCA)"]:::validation
    end
    click VAL_PCA "https://github.com/leo2105/jointmultiunimodalattention/blob/main/validation.py"
    click VAL_noPCA "https://github.com/leo2105/jointmultiunimodalattention/blob/main/validation_nopca.py"
    
    %% Testing Pipeline
    subgraph "Testing Pipeline"
        TEST_PCA["Testing (with PCA)"]:::testing
        TEST_noPCA["Testing (without PCA)"]:::testing
    end
    click TEST_PCA "https://github.com/leo2105/jointmultiunimodalattention/blob/main/testing.py"
    click TEST_noPCA "https://github.com/leo2105/jointmultiunimodalattention/blob/main/testing_nopca.py"
    
    %% Connections
    ENV --> LOSO
    ENV --> NoLOSO
    ENV --> NoPCA

    LOSO --> DL
    NoLOSO --> DL
    NoPCA --> DL

    DL --> PP
    PP --> TP_PCA
    %% For the without PCA branch, data flows directly from Dataset Loader
    DL --> TP_noPCA

    TP_PCA --> MA_PCA
    TP_noPCA --> MA_noPCA

    MA_PCA --> PT

    MA_PCA --> VAL_PCA
    MA_noPCA --> VAL_noPCA

    VAL_PCA --> TEST_PCA
    VAL_noPCA --> TEST_noPCA

    %% Styles
    classDef env fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef protocol fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef data fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef training fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
    classDef model fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef validation fill:#e2e3e5,stroke:#6c757d,stroke-width:2px;
    classDef testing fill:#fefefe,stroke:#818182,stroke-width:2px;
```
