How imadhou/multimodal-sentiment-analysis works
Project type and purpose
• This is a research‐oriented Python machine‐learning pipeline for multimodal sentiment analysis.
• It ingests text and image data, applies attention fusion across modalities, trains models, and evaluates performance.
File structure
Top-level:
• main.py – pipeline orchestrator
• environnement.yml / requirements.txt – Conda/Pip setup
• data/processed – holds preprocessed test images
• notebooks/ – Jupyter notebooks for interactive exploration
• src/ – core code
src/ subdirectories:
• dataset_generators/ – classes to wrap text, image, and multimodal datasets
• models/ – PyTorch model definitions (text_model.py, image_model.py, multi_modal_model.py)
• train/ – training scripts and notebooks
• utils/ – helpers (Emoji_Dict, utility functions)
• images/ – static assets referenced in README
Other:
• .gitignore, LICENSE, README.md
Key insights from the README
• Project goal: fuse text and image features via attention fusion for sentiment classification.
• Core components: Image Model, Text Model, Attention, Fusion Layer, Nonlinear Layer, Classification Layer.
• Training orchestrated via PyTorch, scikit-learn splitting, Adam optimizer, CrossEntropyLoss.
• Data source: T4SA multimodal dataset.
Main components
A. Data Layer
• Raw data (T4SA: text, images, audio placeholders)
• Processed data repository (data/processed)
B. Preprocessing & Dataset Generators
• src/dataset_generators/text_dataset.py
• src/dataset_generators/image_dataset.py
• src/dataset_generators/multi_modal_dataset.py
C. Model Components
• Text Model (src/models/text_model.py)
• Image Model (src/models/image_model.py)
• Attention Fusion & Fusion Layer (src/models/multi_modal_model.py)
• Nonlinear & Classification layers (in multimodal model)
D. Training & Evaluation
• Training scripts (src/train/train_*.py)
• Notebooks for prototyping (notebooks/*.ipynb, src/train/images_model2.ipynb)
• Utils (src/utils/utils.py) – loss/metrics/logging
E. Orchestrator
• main.py – loads environment, triggers data loading, model instantiation, training, evaluation
F. External Dependencies
• Python 3.9, PyTorch, scikit-learn, Conda environment
Relationships and interactions
• Raw data → preprocessing → data/processed
• Preprocessed data → dataset_generators → DataLoader objects
• DataLoader → Model (forward pass) → Attention Fusion → Fused features
• Fused features → Nonlinear → Classification → Predictions
• Predictions + ground truth → Loss function (CrossEntropy) → Backprop (Adam)
• Training loop → model checkpoints → evaluation metrics → result plots (src/images/)
Architectural patterns and design principles
• Separation of concerns: data ingestion, model definitions, training scripts, utilities
• Modularization: each modality has its own dataset and model class
• Reusability: utils.py and dataset_generators serve all training variants
• Experimentation support: Jupyter notebooks coexist with scripted training for prototyping
1. Sketch the data flow from data/ into the dataset_generators package.
2. How each generator is used by the DataLoader (batch size, shuffle).
4. FusionLayer + Nonlinear + Classifier under one Fusion block.
5. Training loop: Loss → Optimizer → Parameter update.
Detail
• File names to map code to architecture
• Both training (src/train/*.py) and orchestration (main.py)
• Evaluation outputs (images/ plots) as terminal module