# nustm/vlp-mabsa architecture

> Architecture diagram of the GitHub repository nustm/vlp-mabsa, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nustm/vlp-mabsa
- Repository: https://github.com/nustm/vlp-mabsa

## Overview

VLP-MABSA is a research system for multimodal aspect-based sentiment analysis. Its documented workflow prepares text spans, sentiment labels, image-region features and ANP annotations; pretrains on multimodal objectives; then fine-tunes on Twitter data for aspect/opinion extraction and aspect sentiment classification. The graph separates data preparation, modeling, and task-specific training. Some source excerpts are partial or absent, so links are limited to documented capabilities and visible orchestration; uncertain internal calls are omitted.

Main entry points: [`src/model/MAESC_model.py`](https://github.com/nustm/vlp-mabsa/blob/master/src/model/MAESC_model.py) (Multimodal task model), [`pretrain.py`](https://github.com/nustm/vlp-mabsa/blob/master/pretrain.py) (Pre-training entry), [`twitter_sc_training.py`](https://github.com/nustm/vlp-mabsa/blob/master/twitter_sc_training.py) (Twitter SC training), [`twitter_ae_training.py`](https://github.com/nustm/vlp-mabsa/blob/master/twitter_ae_training.py) (Twitter AE training)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data and encoding"]
  node_mvsa_data["MVSA dataset<br/>[dataset.py]"]
  node_twitter_data["Twitter datasets<br/>[dataset.py]"]
  node_tokenizer["Condition tokenizer"]
  node_collator["Task collation<br/>[collation.py]"]
  node_metadata["Dataset metadata"]
end

subgraph group_pretrain["Pre-training"]
  node_pretrain_entry["Pre-training entry<br/>[pretrain.py]"]
  node_pretrain_loop["Pre-training loop<br/>[training.py]"]
  node_checkpoint_utils["Training state<br/>[utils.py]"]
end

subgraph group_model["Model and generation"]
  node_model_config["Model configuration<br/>[config.py]"]
  node_multimodal_model["Multimodal task model<br/>[MAESC_model.py]"]
  node_bart_model["BART backbone<br/>[modeling_bart.py]"]
  node_model_modules["Task modules<br/>[modules.py]"]
  node_generator["Sequence generation<br/>[generater.py]"]
end

subgraph group_downstream["Downstream tasks"]
  node_finetune_entry["Twitter SC training"]
  node_ae_entry["Twitter AE training"]
  node_maesc_entry["MAESC training<br/>[MAESC_training.py]"]
  node_finetune_loop["Fine-tuning and evaluation<br/>[training.py]"]
  node_evaluation["Metrics and evaluation<br/>[eval_utils.py]"]
end

node_researcher(("Researcher"))
node_external_resources[("Prepared datasets and features")]

node_researcher -->|"starts pretraining"| node_pretrain_entry
node_researcher -->|"starts sentiment task"| node_finetune_entry
node_researcher -->|"starts extraction task"| node_ae_entry
node_researcher -->|"starts MAESC task"| node_maesc_entry
node_external_resources -->|"provides data"| node_mvsa_data
node_external_resources -->|"provides data"| node_twitter_data
node_metadata -->|"configures inputs"| node_mvsa_data
node_metadata -->|"configures inputs"| node_twitter_data
node_pretrain_entry -->|"loads samples"| node_mvsa_data
node_pretrain_entry -->|"constructs"| node_tokenizer
node_pretrain_entry -->|"constructs"| node_collator
node_pretrain_entry -->|"loads configuration"| node_model_config
node_pretrain_entry -->|"creates model"| node_multimodal_model
node_pretrain_entry -->|"runs epochs"| node_pretrain_loop
node_pretrain_loop -->|"forwards task batches"| node_multimodal_model
node_pretrain_loop -->|"uses training state"| node_checkpoint_utils
node_finetune_entry -->|"loads samples"| node_twitter_data
node_ae_entry -->|"loads samples"| node_twitter_data
node_maesc_entry -->|"loads samples"| node_twitter_data
node_finetune_entry -->|"constructs"| node_tokenizer
node_ae_entry -->|"constructs"| node_tokenizer
node_maesc_entry -->|"constructs"| node_tokenizer
node_finetune_entry -->|"creates model"| node_multimodal_model
node_ae_entry -->|"creates model"| node_multimodal_model
node_maesc_entry -->|"creates model"| node_multimodal_model
node_finetune_entry -->|"runs fine-tuning"| node_finetune_loop
node_ae_entry -->|"runs fine-tuning"| node_finetune_loop
node_maesc_entry -->|"runs fine-tuning"| node_finetune_loop
node_finetune_loop -->|"forwards batches"| node_multimodal_model
node_finetune_loop -->|"evaluates predictions"| node_evaluation
node_multimodal_model -->|"uses backbone"| node_bart_model
node_multimodal_model -->|"uses task modules"| node_model_modules
node_multimodal_model -->|"uses generation"| node_generator
node_multimodal_model -->|"uses configuration"| node_model_config
node_tokenizer -->|"encodes batch inputs"| node_collator

click node_pretrain_entry "https://github.com/nustm/vlp-mabsa/blob/master/pretrain.py"
click node_mvsa_data "https://github.com/nustm/vlp-mabsa/blob/master/src/data/dataset.py"
click node_twitter_data "https://github.com/nustm/vlp-mabsa/blob/master/src/data/dataset.py"
click node_tokenizer "https://github.com/nustm/vlp-mabsa/blob/master/src/data/tokenization_new.py"
click node_collator "https://github.com/nustm/vlp-mabsa/blob/master/src/data/collation.py"
click node_metadata "https://github.com/nustm/vlp-mabsa/tree/master/src/data/jsons"
click node_pretrain_loop "https://github.com/nustm/vlp-mabsa/blob/master/src/training.py"
click node_finetune_entry "https://github.com/nustm/vlp-mabsa/blob/master/twitter_sc_training.py"
click node_ae_entry "https://github.com/nustm/vlp-mabsa/blob/master/twitter_ae_training.py"
click node_maesc_entry "https://github.com/nustm/vlp-mabsa/blob/master/MAESC_training.py"
click node_finetune_loop "https://github.com/nustm/vlp-mabsa/blob/master/src/training.py"
click node_model_config "https://github.com/nustm/vlp-mabsa/blob/master/src/model/config.py"
click node_multimodal_model "https://github.com/nustm/vlp-mabsa/blob/master/src/model/MAESC_model.py"
click node_bart_model "https://github.com/nustm/vlp-mabsa/blob/master/src/model/modeling_bart.py"
click node_model_modules "https://github.com/nustm/vlp-mabsa/blob/master/src/model/modules.py"
click node_generator "https://github.com/nustm/vlp-mabsa/blob/master/src/model/generater.py"
click node_evaluation "https://github.com/nustm/vlp-mabsa/blob/master/src/eval_utils.py"
click node_checkpoint_utils "https://github.com/nustm/vlp-mabsa/blob/master/src/utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_mvsa_data,node_twitter_data,node_tokenizer,node_collator,node_metadata toneBlue
class node_pretrain_entry,node_pretrain_loop,node_checkpoint_utils,node_external_resources toneAmber
class node_model_config,node_multimodal_model,node_bart_model,node_model_modules,node_generator toneMint
class node_finetune_entry,node_ae_entry,node_maesc_entry,node_finetune_loop,node_evaluation toneRose
class node_researcher toneIndigo
```

## Components

### Data and encoding

- [MVSA dataset](https://github.com/nustm/vlp-mabsa/blob/master/src/data/dataset.py): `src/data/dataset.py`
- [Twitter datasets](https://github.com/nustm/vlp-mabsa/blob/master/src/data/dataset.py): `src/data/dataset.py`
- [Condition tokenizer](https://github.com/nustm/vlp-mabsa/blob/master/src/data/tokenization_new.py): `src/data/tokenization_new.py`
- [Task collation](https://github.com/nustm/vlp-mabsa/blob/master/src/data/collation.py): `src/data/collation.py`
- [Dataset metadata](https://github.com/nustm/vlp-mabsa/tree/master/src/data/jsons): `src/data/jsons`

### Pre-training

- [Pre-training entry](https://github.com/nustm/vlp-mabsa/blob/master/pretrain.py): `pretrain.py`
- [Pre-training loop](https://github.com/nustm/vlp-mabsa/blob/master/src/training.py): `src/training.py`
- [Training state](https://github.com/nustm/vlp-mabsa/blob/master/src/utils.py): `src/utils.py`

### Model and generation

- [Model configuration](https://github.com/nustm/vlp-mabsa/blob/master/src/model/config.py): `src/model/config.py`
- [Multimodal task model](https://github.com/nustm/vlp-mabsa/blob/master/src/model/MAESC_model.py): `src/model/MAESC_model.py`
- [BART backbone](https://github.com/nustm/vlp-mabsa/blob/master/src/model/modeling_bart.py): `src/model/modeling_bart.py`
- [Task modules](https://github.com/nustm/vlp-mabsa/blob/master/src/model/modules.py): `src/model/modules.py`
- [Sequence generation](https://github.com/nustm/vlp-mabsa/blob/master/src/model/generater.py): `src/model/generater.py`

### Downstream tasks

- [Twitter SC training](https://github.com/nustm/vlp-mabsa/blob/master/twitter_sc_training.py): `twitter_sc_training.py`
- [Twitter AE training](https://github.com/nustm/vlp-mabsa/blob/master/twitter_ae_training.py): `twitter_ae_training.py`
- [MAESC training](https://github.com/nustm/vlp-mabsa/blob/master/MAESC_training.py): `MAESC_training.py`
- [Fine-tuning and evaluation](https://github.com/nustm/vlp-mabsa/blob/master/src/training.py): `src/training.py`
- [Metrics and evaluation](https://github.com/nustm/vlp-mabsa/blob/master/src/eval_utils.py): `src/eval_utils.py`

### Other components

- **Researcher**
- **Prepared datasets and features**

## Connections

- Researcher → Pre-training entry: starts pretraining
- Researcher → Twitter SC training: starts sentiment task
- Researcher → Twitter AE training: starts extraction task
- Researcher → MAESC training: starts MAESC task
- Prepared datasets and features → MVSA dataset: provides data
- Prepared datasets and features → Twitter datasets: provides data
- Dataset metadata → MVSA dataset: configures inputs
- Dataset metadata → Twitter datasets: configures inputs
- Pre-training entry → MVSA dataset: loads samples
- Pre-training entry → Condition tokenizer: constructs
- Pre-training entry → Task collation: constructs
- Pre-training entry → Model configuration: loads configuration
- Pre-training entry → Multimodal task model: creates model
- Pre-training entry → Pre-training loop: runs epochs
- Pre-training loop → Multimodal task model: forwards task batches
- Pre-training loop → Training state: uses training state
- Twitter SC training → Twitter datasets: loads samples
- Twitter AE training → Twitter datasets: loads samples
- MAESC training → Twitter datasets: loads samples
- Twitter SC training → Condition tokenizer: constructs
- Twitter AE training → Condition tokenizer: constructs
- MAESC training → Condition tokenizer: constructs
- Twitter SC training → Multimodal task model: creates model
- Twitter AE training → Multimodal task model: creates model
- MAESC training → Multimodal task model: creates model
- Twitter SC training → Fine-tuning and evaluation: runs fine-tuning
- Twitter AE training → Fine-tuning and evaluation: runs fine-tuning
- MAESC training → Fine-tuning and evaluation: runs fine-tuning
- Fine-tuning and evaluation → Multimodal task model: forwards batches
- Fine-tuning and evaluation → Metrics and evaluation: evaluates predictions
- Multimodal task model → BART backbone: uses backbone
- Multimodal task model → Task modules: uses task modules
- Multimodal task model → Sequence generation: uses generation
- Multimodal task model → Model configuration: uses configuration
- Condition tokenizer → Task collation: encodes batch inputs