# redaimao/mmdd2025 architecture

> Architecture diagram of the GitHub repository redaimao/mmdd2025, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/redaimao/mmdd2025
- Repository: https://github.com/redaimao/mmdd2025

## Overview

##### Main Components
- Data Loader Layer: Represented by files in the data_loader directory. These scripts (e.g., Load_BagOfLies_face_audio_OpenFace_Affect.py, Load_MU3D_face_audio_wave_OpenFace_Affect.py, etc.) are responsible for loading and preprocessing the extracted features from various datasets.
- Model Definitions: Found in the models directory, these include:
- Specialized networks (e.g., ResNet_Xception.py, pytorch_i3d.py) which indicate that multiple architectures might be used based on the modality.
- The fusion network (fusion_net.py) which combines modality-specific information.
- Additionally, the provided pretrained model (rgb_imagenet.pt) can be depicted as an external asset.
- Modular Components: In the modules directory, you see implementations that support transformer-based strategies such as multihead attention, positional embeddings, and general transformer blocks. These are likely integrated into or augment the model architectures.
- Training & Testing Pipeline: The train_test_feature.py and the accompanying shell script (train_test_feature.sh) serve as orchestrators of the training, validation, and testing processes.
- Utilities & Environment Setup: The utils.py for helper functions and the environment.yml for specifying dependencies also form part of the broader system context.

##### Relationships and Interactions
- Data Flow Starts with the Data Loader:
▸ Extracted features (e.g., from OpenFace and affect data) are read and structured by the data_loader scripts.
- Model Processing:
▸ Separate pathways for different modalities may be visualized, with each modality feeding into its own encoder (like ResNet_Xception or pytorch_i3d) before the fusion network (fusion_net.py) integrates them.
- Training & Testing:
▸ Show a module representing the train_test_feature orchestration where model inputs are fed, loss computed, and weights updated. Illustrate this step as part of the end-to-end pipeline.
- Environment and Dependency Management:
▸ Optionally add a note: “environment.yml defines the runtime and library dependencies,” ensuring that the deployment or local development context is properly encapsulated.

##### Architectural Patterns and Design Principles
- Separation of Concerns: Clearly separate data handling (data_loader), modeling (models and modules), and training orchestration.
- Modularity: Each component (loading, modular networks, fusion, training) is isolated, allowing for easier replacements or updates. This modularity is a key design principle here.
- Fusion Technique Emphasis: Highlight that the system is built around fusion of multiple modalities—a central theme of the project’s architecture.
- Reusability: Components like the transformer modules indicate advanced techniques for capturing inter-modal interactions, emphasizing parameter efficiency and cross-modal learning.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global Input
    DS["Data Source (Datasets)"]:::input

    %% Data Loader Layer
    subgraph "Data Loader Layer"
        DL1["Load_BagOfLies (Data Loader)"]:::data
        DL2["Load_MU3D (Data Loader)"]:::data
    end

    %% Model Definitions
    subgraph "Model Definitions"
        M1["ResNet_Xception"]:::model
        M2["pytorch_i3d"]:::model
        FN["Fusion Network"]:::model
        PM["Pretrained Model"]:::external
    end

    %% Modular Components
    subgraph "Modular Components"
        TC1["Multihead Attention"]:::module
        TC2["Position Embedding"]:::module
        TC3["Transformer"]:::module
    end

    %% Training & Testing Pipeline
    subgraph "Training & Testing Pipeline"
        TT["train_test_feature Pipeline"]:::orchestration
        TS["Shell Script (train_test_feature.sh)"]:::orchestration
    end

    %% Utilities & Environment Setup
    subgraph "Utilities & Environment Setup"
        UT["Utilities"]:::utility
        EN["Environment Setup"]:::utility
    end

    %% Connections: Data Flow
    DS -->|"extracts_features"| DL1
    DS -->|"extracts_features"| DL2

    DL1 -->|"preprocessed_features"| M1
    DL2 -->|"preprocessed_features"| M2

    M1 -->|"feature_stream"| FN
    M2 -->|"feature_stream"| FN

    %% External pretrained model provides weights
    PM -->|"external_weights"| M1

    %% Modular components enhance Fusion Network
    TC1 -->|"enhances"| FN
    TC2 -->|"enhances"| FN
    TC3 -->|"enhances"| FN

    %% Training and Testing orchestration
    FN -->|"model_input"| TT
    TT -->|"triggers"| TS

    %% Utilities support training pipeline
    UT -.-> TT
    EN -.-> TT
    EN -.-> UT

    %% Click Events
    click DL1 "https://github.com/redaimao/mmdd2025/blob/main/data_loader/Load_BagOfLies_face_audio_OpenFace_Affect.py"
    click DL2 "https://github.com/redaimao/mmdd2025/blob/main/data_loader/Load_MU3D_face_audio_OpenFace_Affect.py"
    click M1 "https://github.com/redaimao/mmdd2025/blob/main/models/ResNet_Xception.py"
    click M2 "https://github.com/redaimao/mmdd2025/blob/main/models/pytorch_i3d.py"
    click FN "https://github.com/redaimao/mmdd2025/blob/main/models/fusion_net.py"
    click PM "https://github.com/redaimao/mmdd2025/blob/main/models/rgb_imagenet.pt"
    click TC1 "https://github.com/redaimao/mmdd2025/blob/main/modules/multihead_attention.py"
    click TC2 "https://github.com/redaimao/mmdd2025/blob/main/modules/position_embedding.py"
    click TC3 "https://github.com/redaimao/mmdd2025/blob/main/modules/transformer.py"
    click TT "https://github.com/redaimao/mmdd2025/blob/main/train_test_feature.py"
    click TS "https://github.com/redaimao/mmdd2025/blob/main/train_test_feature.sh"
    click UT "https://github.com/redaimao/mmdd2025/blob/main/utils.py"
    click EN "https://github.com/redaimao/mmdd2025/blob/main/environment.yml"

    %% Style Definitions
    classDef input fill:#f9f,stroke:#333,stroke-width:2px;
    classDef data fill:#bbf,stroke:#333,stroke-width:2px;
    classDef model fill:#bfb,stroke:#333,stroke-width:2px;
    classDef external fill:#ffb,stroke:#333,stroke-width:2px;
    classDef module fill:#fbf,stroke:#333,stroke-width:2px;
    classDef orchestration fill:#ffc,stroke:#333,stroke-width:2px;
    classDef utility fill:#cfc,stroke:#333,stroke-width:2px;
```
