# hoangquy18/multimodal-aspect-category-sentiment-analysis architecture

> Architecture diagram of the GitHub repository hoangquy18/multimodal-aspect-category-sentiment-analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hoangquy18/multimodal-aspect-category-sentiment-analysis
- Repository: https://github.com/hoangquy18/multimodal-aspect-category-sentiment-analysis

## Overview

##### Project Type & Purpose
- The project is a research-oriented multimodal sentiment analysis system.
- It processes paired text-image inputs from the ViMACSA dataset to identify fine-grained aspect categories in the hotel domain.
- The system uses both text preprocessing (leveraging models like “vinai/phobert-base”) and image processing (via category and RoI modules) to fuse modalities with deep neural networks.

##### Analyzing the File Structure
- fcmf_framework folder: Contains core modeling modules:
– fcmf_modeling.py: Likely responsible for the overall fusion modeling.
– mm_modeling.py: Probably handles multimodal modeling specifics.
– roi_modeling.py and image_process.py: Process Regions of Interest and image-specific features.
– optimization.py and resnet_utils.py: Support training optimizations and utility functions (possibly for pre-trained architectures like ResNet).
- image_processing folder: Contains scripts (run_image_categories.py, run_roi_categories.py) that run the image and region proposals pipelines.
- Other scripts (inference.py, run_multimodal_fcmf.py, text_preprocess.py, vimacsa_dataset.py) point to data preparation, the main training loop (including multi-GPU training via torchrun and DDP), and inference routines.
- The README offers commands illustrating how to train, process images/ROIs, and run inference, indicating a setup that links data ingestion, preprocessing, fusion modeling, and prediction.

##### Core Components
a. Data Sources:
– ViMACSA Dataset: Text-image pairs with annotations.
– External weights/models (e.g., pretrained “vinai/phobert-base” for text processing, and potentially ResNet or similar for image features).
b. Preprocessing Modules:
– Text Preprocessing: Managed by text_preprocess.py and vimacsa_dataset.py.
– Image Preprocessing: Handled by image_processing scripts (run_image_categories.py, run_roi_categories.py) and related utility processing in fcmf_framework/image_process.py.
c. Fusion & Modeling Components (within fcmf_framework):
– Text Modeling: Embedding extraction and processing of text data.
– Image/ROI Modeling: Extracting features from images and regions via image_process.py, roi_modeling.py, and resnet_utils.py.
– Multimodal Fusion: Integrated in fcmf_modeling.py and mm_modeling.py, merging text and image features.
– Optimization & Training: optimization.py, usage of torchrun for distributed training, and configuration through run_multimodal_fcmf.py.
d. Inference & Output:
– The inference.py script orchestrates final predictions by accepting text and image inputs.

##### Architectural Patterns & Design Principles
- Modular Design: Split between preprocessing, modeling, and inference.
- Separation of Concerns: Clear separation between text and image pipelines before fusion.
- Distributed Training: Use of DDP (Distributed Data Parallel) as indicated by torchrun for scalability.
- Extensibility: The design appears flexible for adding extra modalities (future integration with Spark is also on the roadmap).

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["ViMACSA Dataset: 4,876 text-image pairs"]:::dataset
    subgraph "Text Pipeline"
        A1["Text Preprocessing"]:::processing
    end
    subgraph "Image Pipeline"
        A2["Image Preprocessing (Categories)"]:::processing
        A3["Image Preprocessing (RoI Extraction)"]:::processing
        A4["Image Processing Utilities"]:::processing
        A5["ROI Modeling for Images"]:::modeling
        A6["ResNet Utilities"]:::processing
    end
    F["Fusion Modeling (FCMF)"]:::fusion
    M["Multimodal Modeling Components"]:::fusion
    subgraph "Training"
        O1["Optimization & Distributed Training (torchrun, DDP)"]:::train
        O2["Main Training Script"]:::train
    end
    INF["Inference Pipeline"]:::inference
    EP["Phobert Pretrained"]:::external
    ER["ResNet Pretrained"]:::external

    A -->|"dataflow"| A1
    A -->|"dataflow"| A2
    A -->|"dataflow"| A3
    A1 -->|"features"| F
    A2 -->|"extracts"| A4
    A3 -->|"extracts"| A4
    A4 -->|"processes"| A5
    A4 -->|"processes"| A6
    A5 -->|"image features"| F
    A6 -->|"image features"| F
    EP -->|"provides"| F
    ER -->|"supplies"| A6
    F -->|"fuses modalities"| M
    M -->|"triggers"| O1
    O1 -->|"executes"| O2
    M -->|"feeds"| INF

    click A "https://github.com/hoangquy18/multimodal-aspect-category-sentiment-analysis/blob/main/vimacsa_dataset.py"
    click A1 "https://github.com/hoangquy18/multimodal-aspect-category-sentiment-analysis/blob/main/text_preprocess.py"
    click A2 "https://github.com/hoangquy18/multimodal-aspect-category-sentiment-analysis/blob/main/image_processing/run_image_categories.py"
    click A3 "https://github.com/hoangquy18/multimodal-aspect-category-sentiment-analysis/blob/main/image_processing/run_roi_categories.py"
    click A4 "https://github.com/hoangquy18/multimodal-aspect-category-sentiment-analysis/blob/main/fcmf_framework/image_process.py"
    click A5 "https://github.com/hoangquy18/multimodal-aspect-category-sentiment-analysis/blob/main/fcmf_framework/roi_modeling.py"
    click A6 "https://github.com/hoangquy18/multimodal-aspect-category-sentiment-analysis/blob/main/fcmf_framework/resnet_utils.py"
    click F "https://github.com/hoangquy18/multimodal-aspect-category-sentiment-analysis/blob/main/fcmf_framework/fcmf_modeling.py"
    click M "https://github.com/hoangquy18/multimodal-aspect-category-sentiment-analysis/blob/main/fcmf_framework/mm_modeling.py"
    click O1 "https://github.com/hoangquy18/multimodal-aspect-category-sentiment-analysis/blob/main/fcmf_framework/optimization.py"
    click O2 "https://github.com/hoangquy18/multimodal-aspect-category-sentiment-analysis/blob/main/run_multimodal_fcmf.py"
    click INF "https://github.com/hoangquy18/multimodal-aspect-category-sentiment-analysis/blob/main/inference.py"

    classDef dataset fill:#FFEEAD,stroke:#333,stroke-width:2px;
    classDef processing fill:#B3E5FC,stroke:#333,stroke-width:2px;
    classDef modeling fill:#C8E6C9,stroke:#333,stroke-width:2px;
    classDef fusion fill:#D1C4E9,stroke:#333,stroke-width:2px;
    classDef train fill:#FFCDD2,stroke:#333,stroke-width:2px;
    classDef inference fill:#F0F4C3,stroke:#333,stroke-width:2px;
    classDef external fill:#FFE0B2,stroke:#333,stroke-width:2px;
```
