How qinyang79/ibml works
IBML is a PyTorch research framework for information-balanced multimodal classification across audio-visual, image-text, and 2D-3D tasks. The user-facing workflow is to select a task, provide its paired-modality dataset, train or evaluate a task-specific classifier, and inspect predictions and metrics. The README describes BIO and TCM as framework capabilities, but the sampled excerpts do not establish their implementation wiring, so those links are not drawn. Some task-to-module details are inferred only where the tree identifies corresponding modules; uncertain internal calls are omitted.
Main entry points: main_ModelNet.py ModelNet run · main_AV.py Audio-visual run · main_VL.py Image-text run · main_3DMNIST.py 3D MNIST run
Components 20
Task entrypoints
- Audio-visual run
main_AV.py - Image-text run
main_VL.py - 3D MNIST run
main_3DMNIST.py - ModelNet run
main_ModelNet.py
Multimodal data
- Audio-visual samples
dataset/AVdataset.py - Image-text samples
dataset/VLdataset.py - MNIST samples
dataset/Mnistdataset.py - ModelNet train data
dataset/ModelNet40trainset.py - ModelNet test data
dataset/ModelNet40testset.py
Models and fusion
- Audio-visual classifier
models/basic_model_AV.py - Image-text classifier
models/basic_model_VL.py - 2D-3D classifier
models/basic_model_2D3D.py - Audio-visual backbone
models/backbone_AV.py - Vision-language backbone
models/backbone_VL.py - Point-image backbone
models/backbone_2D3D.py - Fusion modules
models/fusion_modules.py - GloVe encoder
utils/glove_encoder.py
Evaluation and results
- Evaluation metrics
utils/evaluation.py - Saved checkpoints
Other components
- Researcher
Connections 28
- Researcher → Audio-visual run: selects task
- Researcher → Image-text run: selects task
- Researcher → 3D MNIST run: selects task
- Researcher → ModelNet run: selects task
- Audio-visual run → Audio-visual samples: loads batches
- Audio-visual run → Audio-visual classifier: trains or evaluates
- Image-text run → Image-text samples: loads batches
- Image-text run → Image-text classifier: trains or evaluates
- 3D MNIST run → MNIST samples: loads batches
- 3D MNIST run → 2D-3D classifier: trains or evaluates
- ModelNet run → ModelNet train data: loads training data
- ModelNet run → ModelNet test data: loads test data
- ModelNet run → 2D-3D classifier: trains or evaluates
- Audio-visual classifier → Audio-visual backbone: extracts features
- Audio-visual classifier → Fusion modules: fuses modalities
- Image-text classifier → Vision-language backbone: extracts features
- Image-text classifier → GloVe encoder: encodes text
- Image-text classifier → Fusion modules: fuses modalities
- 2D-3D classifier → Point-image backbone: extracts features
- 2D-3D classifier → Fusion modules: fuses modalities
- Audio-visual run → Evaluation metrics: evaluates predictions
- Image-text run → Evaluation metrics: evaluates predictions
- 3D MNIST run → Evaluation metrics: evaluates predictions
- ModelNet run → Evaluation metrics: evaluates predictions
- Audio-visual run → Saved checkpoints: saves weights
- Image-text run → Saved checkpoints: saves weights
- 3D MNIST run → Saved checkpoints: saves weights
- ModelNet run → Saved checkpoints: saves weights