How hoangquy18/multimodal-aspect-category-sentiment-analysis works
Project Type & Purpose
• The project is a research-oriented multimodal sentiment analysis system.
• It processes paired text-image inputs from the ViMACSA dataset to identify fine-grained aspect categories in the hotel domain.
• The system uses both text preprocessing (leveraging models like “vinai/phobert-base”) and image processing (via category and RoI modules) to fuse modalities with deep neural networks.
Analyzing the File Structure
• fcmf_framework folder: Contains core modeling modules:
– fcmf_modeling.py: Likely responsible for the overall fusion modeling.
– mm_modeling.py: Probably handles multimodal modeling specifics.
– roi_modeling.py and image_process.py: Process Regions of Interest and image-specific features.
– optimization.py and resnet_utils.py: Support training optimizations and utility functions (possibly for pre-trained architectures like ResNet).
• image_processing folder: Contains scripts (run_image_categories.py, run_roi_categories.py) that run the image and region proposals pipelines.
• Other scripts (inference.py, run_multimodal_fcmf.py, text_preprocess.py, vimacsa_dataset.py) point to data preparation, the main training loop (including multi-GPU training via torchrun and DDP), and inference routines.
• The README offers commands illustrating how to train, process images/ROIs, and run inference, indicating a setup that links data ingestion, preprocessing, fusion modeling, and prediction.
Core Components
a. Data Sources:
– ViMACSA Dataset: Text-image pairs with annotations.
– External weights/models (e.g., pretrained “vinai/phobert-base” for text processing, and potentially ResNet or similar for image features).
b. Preprocessing Modules:
– Text Preprocessing: Managed by text_preprocess.py and vimacsa_dataset.py.
– Image Preprocessing: Handled by image_processing scripts (run_image_categories.py, run_roi_categories.py) and related utility processing in fcmf_framework/image_process.py.
c. Fusion & Modeling Components (within fcmf_framework):
– Text Modeling: Embedding extraction and processing of text data.
– Image/ROI Modeling: Extracting features from images and regions via image_process.py, roi_modeling.py, and resnet_utils.py.
– Multimodal Fusion: Integrated in fcmf_modeling.py and mm_modeling.py, merging text and image features.
– Optimization & Training: optimization.py, usage of torchrun for distributed training, and configuration through run_multimodal_fcmf.py.
d. Inference & Output:
– The inference.py script orchestrates final predictions by accepting text and image inputs.
Architectural Patterns & Design Principles
• Modular Design: Split between preprocessing, modeling, and inference.
• Separation of Concerns: Clear separation between text and image pipelines before fusion.
• Distributed Training: Use of DDP (Distributed Data Parallel) as indicated by torchrun for scalability.
• Extensibility: The design appears flexible for adding extra modalities (future integration with Spark is also on the roadmap).