# imadhou/multimodal-sentiment-analysis architecture

> Architecture diagram of the GitHub repository imadhou/multimodal-sentiment-analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/imadhou/multimodal-sentiment-analysis
- Repository: https://github.com/imadhou/multimodal-sentiment-analysis

## Overview

##### Project type and purpose
- This is a research‐oriented Python machine‐learning pipeline for multimodal sentiment analysis.
- It ingests text and image data, applies attention fusion across modalities, trains models, and evaluates performance.

##### File structure
Top-level:
- main.py – pipeline orchestrator
- environnement.yml / requirements.txt – Conda/Pip setup
- data/processed – holds preprocessed test images
- notebooks/ – Jupyter notebooks for interactive exploration
- src/ – core code
src/ subdirectories:
- dataset_generators/ – classes to wrap text, image, and multimodal datasets
- models/ – PyTorch model definitions (text_model.py, image_model.py, multi_modal_model.py)
- train/ – training scripts and notebooks
- utils/ – helpers (Emoji_Dict, utility functions)
- images/ – static assets referenced in README
Other:
- .gitignore, LICENSE, README.md

##### Key insights from the README
- Project goal: fuse text and image features via attention fusion for sentiment classification.
- Core components: Image Model, Text Model, Attention, Fusion Layer, Nonlinear Layer, Classification Layer.
- Training orchestrated via PyTorch, scikit-learn splitting, Adam optimizer, CrossEntropyLoss.
- Data source: T4SA multimodal dataset.

##### Main components
A. Data Layer
- Raw data (T4SA: text, images, audio placeholders)
- Processed data repository (data/processed)
B. Preprocessing & Dataset Generators
- src/dataset_generators/text_dataset.py
- src/dataset_generators/image_dataset.py
- src/dataset_generators/multi_modal_dataset.py
C. Model Components
- Text Model (src/models/text_model.py)
- Image Model (src/models/image_model.py)
- Attention Fusion & Fusion Layer (src/models/multi_modal_model.py)
- Nonlinear & Classification layers (in multimodal model)
D. Training & Evaluation
- Training scripts (src/train/train_*.py)
- Notebooks for prototyping (notebooks/*.ipynb, src/train/images_model2.ipynb)
- Utils (src/utils/utils.py) – loss/metrics/logging
E. Orchestrator
- main.py – loads environment, triggers data loading, model instantiation, training, evaluation
F. External Dependencies
- Python 3.9, PyTorch, scikit-learn, Conda environment

##### Relationships and interactions
- Raw data → preprocessing → data/processed
- Preprocessed data → dataset_generators → DataLoader objects
- DataLoader → Model (forward pass) → Attention Fusion → Fused features
- Fused features → Nonlinear → Classification → Predictions
- Predictions + ground truth → Loss function (CrossEntropy) → Backprop (Adam)
- Training loop → model checkpoints → evaluation metrics → result plots (src/images/)

##### Architectural patterns and design principles
- Separation of concerns: data ingestion, model definitions, training scripts, utilities
- Modularization: each modality has its own dataset and model class
- Reusability: utils.py and dataset_generators serve all training variants
- Experimentation support: Jupyter notebooks coexist with scripted training for prototyping

1. Sketch the data flow from `data/` into the `dataset_generators` package.
2. How each generator is used by the DataLoader (batch size, shuffle).
4. FusionLayer + Nonlinear + Classifier under one Fusion block.
5. Training loop: Loss → Optimizer → Parameter update.

##### Detail
- File names to map code to architecture
- Both training (src/train/*.py) and orchestration (main.py)
- Evaluation outputs (images/ plots) as terminal module

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    RawData["T4SA Raw Data\n(Text, Images, Audio)"]:::data-store
    ProcessedData["Processed Data Repository"]:::data-store
    TestImages["Raw Test Images"]:::data-store

    RawData -->|"preprocess()"| ProcessedData
    ProcessedData -->|"loads test_images"| TestImages

    %% Preprocessing & Dataset Generators
    subgraph "Preprocessing & Dataset Generators"
        direction TB
        TextDS["Text Dataset Generator"]:::module
        ImageDS["Image Dataset Generator"]:::module
        MultiDS["Multimodal Dataset Generator"]:::module
    end

    ProcessedData -->|"/__getitem__()"| TextDS
    ProcessedData -->|"/__getitem__()"| ImageDS
    TextDS -->|"batch data"| MultiDS
    ImageDS -->|"batch data"| MultiDS

    %% Model Components
    subgraph "Model Components"
        direction TB
        TextModel["Text Model"]:::module
        ImageModel["Image Model"]:::module
        AttentionFusion["Attention Fusion & Fusion Layer"]:::module
    end

    TextDS -->|"forward()"| TextModel
    ImageDS -->|"forward()"| ImageModel
    TextModel -->|"features"| AttentionFusion
    ImageModel -->|"features"| AttentionFusion

    %% Training & Evaluation
    subgraph "Training & Evaluation"
        direction TB
        TrainLoop["Training Loop\n(Loss & Optimizer)"]:::script
        EvalPlots["Evaluation Plots\n(src/images)"]:::external
    end

    AttentionFusion -->|"fused features"| TrainLoop
    TrainLoop -->|"update params"| AttentionFusion
    TrainLoop -->|"generate plots"| EvalPlots

    %% Orchestrator & Configs
    subgraph "Orchestrator & Config"
        direction TB
        Main["Pipeline Orchestrator\n(main.py)"]:::script
        Conda["Conda Environment\nenvironnement.yml"]:::external
        PipReq["Pip Dependencies\nrequirements.txt"]:::external
    end

    Main -->|"invokes preprocess"| ProcessedData
    Main -->|"invokes train"| TrainLoop
    Main -->|"loads env"| Conda
    Main -->|"loads deps"| PipReq

    %% Notebooks & Utils
    subgraph "Auxiliary Components"
        direction TB
        NBText["Text Model Notebook"]:::notebook
        NBImage["Image Model Notebook"]:::notebook
        NBMulti["Multimodal Model Notebook"]:::notebook
        NBImage2["Image Model v2 Notebook"]:::notebook
        Utils["Utility Functions"]:::util
        Emoji["Emoji Dictionary"]:::util
        README["README & LICENSE"]:::external
    end

    ProcessedData -.->|dashed| NBText
    ProcessedData -.->|dashed| NBImage
    ProcessedData -.->|dashed| NBMulti
    TrainLoop -.->|dashed| NBImage2
    TrainLoop -.->|dashed| Utils
    TrainLoop -.->|dashed| Emoji
    Main -.->|dashed| README

    %% Click Events
    click ProcessedData "https://github.com/imadhou/multimodal-sentiment-analysis/tree/main/data/processed"
    click TestImages "https://github.com/imadhou/multimodal-sentiment-analysis/blob/main/data/processed/test_images/.test_images"
    click Main "https://github.com/imadhou/multimodal-sentiment-analysis/blob/main/main.py"
    click Conda "https://github.com/imadhou/multimodal-sentiment-analysis/blob/main/environnement.yml"
    click PipReq "https://github.com/imadhou/multimodal-sentiment-analysis/blob/main/requirements.txt"
    click NBText "https://github.com/imadhou/multimodal-sentiment-analysis/blob/main/notebooks/text_model.ipynb"
    click NBImage "https://github.com/imadhou/multimodal-sentiment-analysis/blob/main/notebooks/images_model.ipynb"
    click NBMulti "https://github.com/imadhou/multimodal-sentiment-analysis/blob/main/notebooks/multi_modal_model.ipynb"
    click TextDS "https://github.com/imadhou/multimodal-sentiment-analysis/blob/main/src/dataset_generators/text_dataset.py"
    click ImageDS "https://github.com/imadhou/multimodal-sentiment-analysis/blob/main/src/dataset_generators/image_dataset.py"
    click MultiDS "https://github.com/imadhou/multimodal-sentiment-analysis/blob/main/src/dataset_generators/multi_modal_dataset.py"
    click TextModel "https://github.com/imadhou/multimodal-sentiment-analysis/blob/main/src/models/text_model.py"
    click ImageModel "https://github.com/imadhou/multimodal-sentiment-analysis/blob/main/src/models/image_model.py"
    click AttentionFusion "https://github.com/imadhou/multimodal-sentiment-analysis/blob/main/src/models/multi_modal_model.py"
    click TrainText "https://github.com/imadhou/multimodal-sentiment-analysis/blob/main/src/train/train_text_model.py"
    click TrainImage "https://github.com/imadhou/multimodal-sentiment-analysis/blob/main/src/train/train_image_model.py"
    click TrainMulti "https://github.com/imadhou/multimodal-sentiment-analysis/blob/main/src/train/train_multi_modal_model.py"
    click NBImage2 "https://github.com/imadhou/multimodal-sentiment-analysis/blob/main/src/train/images_model2.ipynb"
    click Utils "https://github.com/imadhou/multimodal-sentiment-analysis/blob/main/src/utils/utils.py"
    click Emoji "https://github.com/imadhou/multimodal-sentiment-analysis/blob/main/src/utils/Emoji_Dict.p"
    click EvalPlots "https://github.com/imadhou/multimodal-sentiment-analysis/tree/main/src/images"
    click README "https://github.com/imadhou/multimodal-sentiment-analysis/blob/main/README.md"
    click README "https://github.com/imadhou/multimodal-sentiment-analysis/tree/main/LICENSE"

    %% Styles
    classDef data-store fill:#C3E6FF,stroke:#2B7CE9,color:#000
    classDef module fill:#DFFFD6,stroke:#27A844,color:#000
    classDef script fill:#FFF5BA,stroke:#D4A11A,color:#000
    classDef notebook fill:#FFE6F0,stroke:#D95587,color:#000
    classDef util fill:#E8E8E8,stroke:#A0A0A0,color:#000
    classDef external fill:#F0F0F0,stroke:#888,color:#000
```
