# qinyang79/ibml architecture

> Architecture diagram of the GitHub repository qinyang79/ibml, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/qinyang79/ibml
- Repository: https://github.com/qinyang79/ibml

## Overview

IBML is a PyTorch research framework for information-balanced multimodal classification across audio-visual, image-text, and 2D-3D tasks. The user-facing workflow is to select a task, provide its paired-modality dataset, train or evaluate a task-specific classifier, and inspect predictions and metrics. The README describes BIO and TCM as framework capabilities, but the sampled excerpts do not establish their implementation wiring, so those links are not drawn. Some task-to-module details are inferred only where the tree identifies corresponding modules; uncertain internal calls are omitted.

Main entry points: [`main_ModelNet.py`](https://github.com/qinyang79/ibml/blob/main/main_ModelNet.py) (ModelNet run), [`main_AV.py`](https://github.com/qinyang79/ibml/blob/main/main_AV.py) (Audio-visual run), [`main_VL.py`](https://github.com/qinyang79/ibml/blob/main/main_VL.py) (Image-text run), [`main_3DMNIST.py`](https://github.com/qinyang79/ibml/blob/main/main_3DMNIST.py) (3D MNIST run)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Task entrypoints"]
  node_av_run["Audio-visual run<br/>[main_AV.py]"]
  node_vl_run["Image-text run<br/>[main_VL.py]"]
  node_mnist_run["3D MNIST run<br/>[main_3DMNIST.py]"]
  node_modelnet_run["ModelNet run<br/>[main_ModelNet.py]"]
end

subgraph group_data["Multimodal data"]
  node_av_data["Audio-visual samples<br/>[AVdataset.py]"]
  node_vl_data["Image-text samples<br/>[VLdataset.py]"]
  node_mnist_data["MNIST samples<br/>[Mnistdataset.py]"]
  node_modelnet_train["ModelNet train data"]
  node_modelnet_test["ModelNet test data"]
end

subgraph group_models["Models and fusion"]
  node_av_model["Audio-visual classifier<br/>[basic_model_AV.py]"]
  node_vl_model["Image-text classifier<br/>[basic_model_VL.py]"]
  node_mnist_model["2D-3D classifier"]
  node_av_backbone["Audio-visual backbone<br/>[backbone_AV.py]"]
  node_vl_backbone["Vision-language backbone<br/>[backbone_VL.py]"]
  node_point_image["Point-image backbone<br/>[backbone_2D3D.py]"]
  node_fusion["Fusion modules<br/>[fusion_modules.py]"]
  node_glove["GloVe encoder<br/>[glove_encoder.py]"]
end

subgraph group_results["Evaluation and results"]
  node_metrics["Evaluation metrics<br/>[evaluation.py]"]
  node_checkpoint[("Saved checkpoints")]
end

node_researcher(("Researcher"))

node_researcher -->|"selects task"| node_av_run
node_researcher -->|"selects task"| node_vl_run
node_researcher -->|"selects task"| node_mnist_run
node_researcher -->|"selects task"| node_modelnet_run
node_av_run -->|"loads batches"| node_av_data
node_av_run -->|"trains or evaluates"| node_av_model
node_vl_run -->|"loads batches"| node_vl_data
node_vl_run -->|"trains or evaluates"| node_vl_model
node_mnist_run -->|"loads batches"| node_mnist_data
node_mnist_run -->|"trains or evaluates"| node_mnist_model
node_modelnet_run -->|"loads training data"| node_modelnet_train
node_modelnet_run -->|"loads test data"| node_modelnet_test
node_modelnet_run -->|"trains or evaluates"| node_mnist_model
node_av_model -->|"extracts features"| node_av_backbone
node_av_model -->|"fuses modalities"| node_fusion
node_vl_model -->|"extracts features"| node_vl_backbone
node_vl_model -.->|"encodes text"| node_glove
node_vl_model -->|"fuses modalities"| node_fusion
node_mnist_model -->|"extracts features"| node_point_image
node_mnist_model -->|"fuses modalities"| node_fusion
node_av_run -->|"evaluates predictions"| node_metrics
node_vl_run -->|"evaluates predictions"| node_metrics
node_mnist_run -->|"evaluates predictions"| node_metrics
node_modelnet_run -->|"evaluates predictions"| node_metrics
node_av_run -->|"saves weights"| node_checkpoint
node_vl_run -->|"saves weights"| node_checkpoint
node_mnist_run -->|"saves weights"| node_checkpoint
node_modelnet_run -->|"saves weights"| node_checkpoint

click node_av_run "https://github.com/qinyang79/ibml/blob/main/main_AV.py"
click node_vl_run "https://github.com/qinyang79/ibml/blob/main/main_VL.py"
click node_mnist_run "https://github.com/qinyang79/ibml/blob/main/main_3DMNIST.py"
click node_modelnet_run "https://github.com/qinyang79/ibml/blob/main/main_ModelNet.py"
click node_av_data "https://github.com/qinyang79/ibml/blob/main/dataset/AVdataset.py"
click node_vl_data "https://github.com/qinyang79/ibml/blob/main/dataset/VLdataset.py"
click node_mnist_data "https://github.com/qinyang79/ibml/blob/main/dataset/Mnistdataset.py"
click node_modelnet_train "https://github.com/qinyang79/ibml/blob/main/dataset/ModelNet40trainset.py"
click node_modelnet_test "https://github.com/qinyang79/ibml/blob/main/dataset/ModelNet40testset.py"
click node_av_model "https://github.com/qinyang79/ibml/blob/main/models/basic_model_AV.py"
click node_vl_model "https://github.com/qinyang79/ibml/blob/main/models/basic_model_VL.py"
click node_mnist_model "https://github.com/qinyang79/ibml/blob/main/models/basic_model_2D3D.py"
click node_av_backbone "https://github.com/qinyang79/ibml/blob/main/models/backbone_AV.py"
click node_vl_backbone "https://github.com/qinyang79/ibml/blob/main/models/backbone_VL.py"
click node_point_image "https://github.com/qinyang79/ibml/blob/main/models/backbone_2D3D.py"
click node_fusion "https://github.com/qinyang79/ibml/blob/main/models/fusion_modules.py"
click node_glove "https://github.com/qinyang79/ibml/blob/main/utils/glove_encoder.py"
click node_metrics "https://github.com/qinyang79/ibml/blob/main/utils/evaluation.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_av_run,node_vl_run,node_mnist_run,node_modelnet_run toneBlue
class node_av_data,node_vl_data,node_mnist_data,node_modelnet_train,node_modelnet_test toneAmber
class node_av_model,node_vl_model,node_mnist_model,node_av_backbone,node_vl_backbone,node_point_image,node_fusion,node_glove toneMint
class node_metrics,node_checkpoint toneRose
class node_researcher toneIndigo
```

## Components

### Task entrypoints

- [Audio-visual run](https://github.com/qinyang79/ibml/blob/main/main_AV.py): `main_AV.py`
- [Image-text run](https://github.com/qinyang79/ibml/blob/main/main_VL.py): `main_VL.py`
- [3D MNIST run](https://github.com/qinyang79/ibml/blob/main/main_3DMNIST.py): `main_3DMNIST.py`
- [ModelNet run](https://github.com/qinyang79/ibml/blob/main/main_ModelNet.py): `main_ModelNet.py`

### Multimodal data

- [Audio-visual samples](https://github.com/qinyang79/ibml/blob/main/dataset/AVdataset.py): `dataset/AVdataset.py`
- [Image-text samples](https://github.com/qinyang79/ibml/blob/main/dataset/VLdataset.py): `dataset/VLdataset.py`
- [MNIST samples](https://github.com/qinyang79/ibml/blob/main/dataset/Mnistdataset.py): `dataset/Mnistdataset.py`
- [ModelNet train data](https://github.com/qinyang79/ibml/blob/main/dataset/ModelNet40trainset.py): `dataset/ModelNet40trainset.py`
- [ModelNet test data](https://github.com/qinyang79/ibml/blob/main/dataset/ModelNet40testset.py): `dataset/ModelNet40testset.py`

### Models and fusion

- [Audio-visual classifier](https://github.com/qinyang79/ibml/blob/main/models/basic_model_AV.py): `models/basic_model_AV.py`
- [Image-text classifier](https://github.com/qinyang79/ibml/blob/main/models/basic_model_VL.py): `models/basic_model_VL.py`
- [2D-3D classifier](https://github.com/qinyang79/ibml/blob/main/models/basic_model_2D3D.py): `models/basic_model_2D3D.py`
- [Audio-visual backbone](https://github.com/qinyang79/ibml/blob/main/models/backbone_AV.py): `models/backbone_AV.py`
- [Vision-language backbone](https://github.com/qinyang79/ibml/blob/main/models/backbone_VL.py): `models/backbone_VL.py`
- [Point-image backbone](https://github.com/qinyang79/ibml/blob/main/models/backbone_2D3D.py): `models/backbone_2D3D.py`
- [Fusion modules](https://github.com/qinyang79/ibml/blob/main/models/fusion_modules.py): `models/fusion_modules.py`
- [GloVe encoder](https://github.com/qinyang79/ibml/blob/main/utils/glove_encoder.py): `utils/glove_encoder.py`

### Evaluation and results

- [Evaluation metrics](https://github.com/qinyang79/ibml/blob/main/utils/evaluation.py): `utils/evaluation.py`
- **Saved checkpoints**

### Other components

- **Researcher**

## Connections

- Researcher → Audio-visual run: selects task
- Researcher → Image-text run: selects task
- Researcher → 3D MNIST run: selects task
- Researcher → ModelNet run: selects task
- Audio-visual run → Audio-visual samples: loads batches
- Audio-visual run → Audio-visual classifier: trains or evaluates
- Image-text run → Image-text samples: loads batches
- Image-text run → Image-text classifier: trains or evaluates
- 3D MNIST run → MNIST samples: loads batches
- 3D MNIST run → 2D-3D classifier: trains or evaluates
- ModelNet run → ModelNet train data: loads training data
- ModelNet run → ModelNet test data: loads test data
- ModelNet run → 2D-3D classifier: trains or evaluates
- Audio-visual classifier → Audio-visual backbone: extracts features
- Audio-visual classifier → Fusion modules: fuses modalities
- Image-text classifier → Vision-language backbone: extracts features
- Image-text classifier → GloVe encoder: encodes text
- Image-text classifier → Fusion modules: fuses modalities
- 2D-3D classifier → Point-image backbone: extracts features
- 2D-3D classifier → Fusion modules: fuses modalities
- Audio-visual run → Evaluation metrics: evaluates predictions
- Image-text run → Evaluation metrics: evaluates predictions
- 3D MNIST run → Evaluation metrics: evaluates predictions
- ModelNet run → Evaluation metrics: evaluates predictions
- Audio-visual run → Saved checkpoints: saves weights
- Image-text run → Saved checkpoints: saves weights
- 3D MNIST run → Saved checkpoints: saves weights
- ModelNet run → Saved checkpoints: saves weights