# vrinda072/mvsa-multimodal-sentiment architecture

> Architecture diagram of the GitHub repository vrinda072/mvsa-multimodal-sentiment, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vrinda072/mvsa-multimodal-sentiment
- Repository: https://github.com/vrinda072/mvsa-multimodal-sentiment

## Overview

MVSA classifies text-and-image posts as negative, neutral, or positive. Its research workflow cleans and labels the dataset, creates train/validation/test splits and features, then compares classifiers and fusion strategies; a follow-up model is saved for inference. Users submit a caption, image, or both through the CLI or local demo and receive probabilities and, in the demo, an explanation of intermediate values and modality agreement. The graph reflects documented notebook stages and sampled runtime code; unsampled notebook details are represented only at their documented subsystem level.

Main entry points: [`src/mvsa/predict.py`](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/src/mvsa/predict.py) (Sentiment inference), [`notebooks/02_modeling/01_training_and_model_selection.ipynb`](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/notebooks/02_modeling/01_training_and_model_selection.ipynb) (Model selection), [`app/server.py`](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/app/server.py) (Demo server), [`src/mvsa/modeling.py`](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/src/mvsa/modeling.py) (Classifiers and fusion)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_dataset[("MVSA-Single data<br/>[README.md]")]
  node_cleaning["Cleaning and labels"]
  node_splitting["Dataset splits"]
  node_features["Feature engineering"]
  node_preprocessing["Preprocessing helpers<br/>[preprocessing.py]"]
  node_text["Text cleaning<br/>[text.py]"]
end

subgraph group_research["Model research"]
  node_modeling["Classifiers and fusion<br/>[modeling.py]"]
  node_model_selection["Model selection"]
  node_robustness["Robustness analysis"]
  node_regularisation["Bias and regularisation"]
  node_error_analysis["Explanations and errors"]
  node_followup_selection["Follow-up selection"]
  node_final_training["Final model training"]
end

subgraph group_inference["Prediction and demo"]
  node_cli["Prediction CLI<br/>[predict.py]"]
  node_sentiment_model["Sentiment inference<br/>[predict.py]"]
  node_demo_page["Local demo page<br/>[index.html]"]
  node_server["Demo server<br/>[server.py]"]
  node_explanations["Prediction explanations<br/>[explain.py]"]
end

subgraph group_artifacts["Evaluation and artifacts"]
  node_state[("Notebook state<br/>[state.py]")]
  node_evaluation["Evaluation tools<br/>[evaluation.py]"]
  node_saved_model[("Saved prediction model<br/>[final_model.npz]")]
  node_explainer_stats[("Explainer statistics")]
end

node_user(("Post author"))

node_dataset -->|"provides data"| node_cleaning
node_cleaning -->|"cleaned posts"| node_splitting
node_splitting -->|"split data"| node_features
node_preprocessing -->|"uses cleaning"| node_text
node_preprocessing -->|"imports helpers"| node_state
node_features -->|"engineered inputs"| node_modeling
node_modeling -->|"supports comparisons"| node_model_selection
node_model_selection -->|"evaluated in"| node_robustness
node_model_selection -->|"analysed in"| node_regularisation
node_model_selection -->|"analysed in"| node_error_analysis
node_modeling -->|"imports helpers"| node_state
node_followup_selection -->|"selects configuration"| node_final_training
node_final_training -->|"uses metrics"| node_evaluation
node_final_training -->|"saves model"| node_saved_model
node_user -->|"submits post"| node_cli
node_cli -->|"calls prediction"| node_sentiment_model
node_user -->|"enters post"| node_demo_page
node_demo_page -->|"sends request"| node_server
node_server -->|"loads model"| node_sentiment_model
node_server -->|"requests explanation"| node_explanations
node_explanations -->|"uses inference"| node_sentiment_model
node_sentiment_model -->|"cleans captions"| node_text
node_sentiment_model -->|"loads weights"| node_saved_model
node_server -.->|"loads statistics"| node_explainer_stats

click node_dataset "https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/data/README.md"
click node_cleaning "https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/notebooks/01_preprocessing/01_data_cleaning_and_labels.ipynb"
click node_splitting "https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/notebooks/01_preprocessing/02_train_val_test_split.ipynb"
click node_features "https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/notebooks/01_preprocessing/03_feature_engineering.ipynb"
click node_preprocessing "https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/src/mvsa/preprocessing.py"
click node_text "https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/src/mvsa/text.py"
click node_modeling "https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/src/mvsa/modeling.py"
click node_model_selection "https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/notebooks/02_modeling/01_training_and_model_selection.ipynb"
click node_robustness "https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/notebooks/02_modeling/02_significance_and_robustness.ipynb"
click node_regularisation "https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/notebooks/02_modeling/03_bias_variance_and_regularisation.ipynb"
click node_error_analysis "https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/notebooks/02_modeling/04_explanation_and_error_analysis.ipynb"
click node_followup_selection "https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/scripts/select_followup_model.py"
click node_final_training "https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/scripts/train_final_model.py"
click node_state "https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/src/mvsa/state.py"
click node_evaluation "https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/src/mvsa/evaluation.py"
click node_saved_model "https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/results/model/final_model.npz"
click node_explainer_stats "https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/results/model/explainer_stats.npz"
click node_cli "https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/scripts/predict.py"
click node_sentiment_model "https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/src/mvsa/predict.py"
click node_demo_page "https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/app/index.html"
click node_server "https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/app/server.py"
click node_explanations "https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/src/mvsa/explain.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dataset,node_cleaning,node_splitting,node_features,node_preprocessing,node_text toneBlue
class node_modeling,node_model_selection,node_robustness,node_regularisation,node_error_analysis,node_followup_selection,node_final_training toneAmber
class node_cli,node_sentiment_model,node_demo_page,node_server,node_explanations toneMint
class node_state,node_evaluation,node_saved_model,node_explainer_stats toneRose
class node_user toneIndigo
```

## Components

### Data preparation

- [MVSA-Single data](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/data/README.md): `data/README.md`
- [Cleaning and labels](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/notebooks/01_preprocessing/01_data_cleaning_and_labels.ipynb): `notebooks/01_preprocessing/01_data_cleaning_and_labels.ipynb`
- [Dataset splits](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/notebooks/01_preprocessing/02_train_val_test_split.ipynb): `notebooks/01_preprocessing/02_train_val_test_split.ipynb`
- [Feature engineering](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/notebooks/01_preprocessing/03_feature_engineering.ipynb): `notebooks/01_preprocessing/03_feature_engineering.ipynb`
- [Preprocessing helpers](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/src/mvsa/preprocessing.py): `src/mvsa/preprocessing.py`
- [Text cleaning](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/src/mvsa/text.py): `src/mvsa/text.py`

### Model research

- [Classifiers and fusion](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/src/mvsa/modeling.py): `src/mvsa/modeling.py`
- [Model selection](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/notebooks/02_modeling/01_training_and_model_selection.ipynb): `notebooks/02_modeling/01_training_and_model_selection.ipynb`
- [Robustness analysis](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/notebooks/02_modeling/02_significance_and_robustness.ipynb): `notebooks/02_modeling/02_significance_and_robustness.ipynb`
- [Bias and regularisation](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/notebooks/02_modeling/03_bias_variance_and_regularisation.ipynb): `notebooks/02_modeling/03_bias_variance_and_regularisation.ipynb`
- [Explanations and errors](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/notebooks/02_modeling/04_explanation_and_error_analysis.ipynb): `notebooks/02_modeling/04_explanation_and_error_analysis.ipynb`
- [Follow-up selection](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/scripts/select_followup_model.py): `scripts/select_followup_model.py`
- [Final model training](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/scripts/train_final_model.py): `scripts/train_final_model.py`

### Prediction and demo

- [Prediction CLI](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/scripts/predict.py): `scripts/predict.py`
- [Sentiment inference](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/src/mvsa/predict.py): `src/mvsa/predict.py`
- [Local demo page](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/app/index.html): `app/index.html`
- [Demo server](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/app/server.py): `app/server.py`
- [Prediction explanations](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/src/mvsa/explain.py): `src/mvsa/explain.py`

### Evaluation and artifacts

- [Notebook state](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/src/mvsa/state.py): `src/mvsa/state.py`
- [Evaluation tools](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/src/mvsa/evaluation.py): `src/mvsa/evaluation.py`
- [Saved prediction model](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/results/model/final_model.npz): `results/model/final_model.npz`
- [Explainer statistics](https://github.com/vrinda072/mvsa-multimodal-sentiment/blob/main/results/model/explainer_stats.npz): `results/model/explainer_stats.npz`

### Other components

- **Post author**

## Connections

- MVSA-Single data → Cleaning and labels: provides data (evidence: `README.md`)
- Cleaning and labels → Dataset splits: cleaned posts (evidence: `README.md`)
- Dataset splits → Feature engineering: split data (evidence: `README.md`)
- Preprocessing helpers → Text cleaning: uses cleaning (evidence: `src/mvsa/preprocessing.py`)
- Preprocessing helpers → Notebook state: imports helpers (evidence: `src/mvsa/preprocessing.py`)
- Feature engineering → Classifiers and fusion: engineered inputs (evidence: `README.md`)
- Classifiers and fusion → Model selection: supports comparisons (evidence: `README.md`)
- Model selection → Robustness analysis: evaluated in (evidence: `README.md`)
- Model selection → Bias and regularisation: analysed in (evidence: `README.md`)
- Model selection → Explanations and errors: analysed in (evidence: `README.md`)
- Classifiers and fusion → Notebook state: imports helpers (evidence: `src/mvsa/modeling.py`)
- Follow-up selection → Final model training: selects configuration (evidence: `scripts/train_final_model.py`)
- Final model training → Evaluation tools: uses metrics (evidence: `scripts/train_final_model.py`)
- Final model training → Saved prediction model: saves model (evidence: `README.md`)
- Post author → Prediction CLI: submits post (evidence: `README.md`)
- Prediction CLI → Sentiment inference: calls prediction (evidence: `scripts/predict.py`)
- Post author → Local demo page: enters post (evidence: `README.md`)
- Local demo page → Demo server: sends request (evidence: `README.md`)
- Demo server → Sentiment inference: loads model (evidence: `app/server.py`)
- Demo server → Prediction explanations: requests explanation (evidence: `app/server.py`)
- Prediction explanations → Sentiment inference: uses inference (evidence: `src/mvsa/explain.py`)
- Sentiment inference → Text cleaning: cleans captions (evidence: `src/mvsa/predict.py`)
- Sentiment inference → Saved prediction model: loads weights (evidence: `src/mvsa/predict.py`)
- Demo server → Explainer statistics: loads statistics
