How xzheng0427/xfmamba works
1. What this repo is
- This is a research training/inference codebase for multi-view medical image classification.
- The core idea is a cross-fusion Mamba model that combines features from two views of the same exam, with dataset-specific pipelines for MURA, CheXpert, and CBIS-DDSM.
2. Top-level execution flow
- 1_train_model.py is the training entry point.
- 2_inference_mura.py, 2_inference_chexpert.py, and 2_inference_ddsmxray.py are dataset-specific inference entry points.
- These scripts likely parse CLI arguments, build the dataset/model, then hand control to shared training or evaluation code under libs/.
3. Data boundary
- The repository expects dataset metadata in CSV form plus image files from the public medical datasets.
- Dataset-specific loaders live in:
- libs/dataset_mura_twoview.py
- libs/dataset_chexpert_twoview.py
- libs/dataset_ddsmxray_twoview.py
- libs/dataset_hipxray_twoview.py
- The repeated twoview naming is important: the model is not just classifying a single image, but consuming paired views as the primary input unit.
4. Preprocessing / spatial handling
- libs/nyu_cropping.py and libs/calc_optimal_centers.py suggest optional image localization or cropping logic before classification.
- This preprocessing is part of the data pipeline rather than the model itself, so it sits on the input side of the architecture.
5. Shared training orchestration
- libs/training.py and libs/train_mutual_learning.py are the main reusable training loops.
- libs/early_stop.py and top-level early_stop.py provide stopping logic.
- libs/utils.py and libs/config.py likely centralize metrics, argument/config handling, logging, and checkpoint utilities.
- This is the main “control plane” separating dataset/model-specific code from generic training mechanics.
6. Model assembly boundary
- net_fusionmamba.py appears to be the main network definition that wires together the backbone and fusion logic.
- models/ contains the model implementations and variants, including:
- models/vmamba.py for the VMamba backbone
- models/fusion_vmamba.py for the cross-view fusion layer
- models/csm_triton.py and models/csms6s.py for alternative Mamba/scan implementations or wrappers
- In the architecture diagram, this is the central compute block between input views and classification head.
7. Backbone and fusion structure
- The project builds on VMamba as the feature extractor, then applies cross-fusion between the two views.
- The README’s model name dualfusionmambav13 hints that the main network is a dual-branch design with feature interaction between branches.
- The output is a task-specific classifier head with configurable num_classes.
8. Mamba / selective scan subsystem
- The models/mamba2/ directory contains low-level Mamba state-space layers and supporting tensor/state update code.
- models/selective_scan/ is a custom C++/CUDA extension with:
- setup.py
- csrc/ kernels
- Python bindings in selective_scan_interface.py
- This is a major architectural boundary: part of the computation runs through compiled GPU kernels, not pure Python.
9. Runtime and external dependencies
- The repo is built around PyTorch 2.2, Triton, CUDA, and the external mamba_ssm wheel.
- The selective-scan kernels are performance-critical infrastructure for the Mamba backbone.
- timm, einops, fvcore, and yacs support model building, tensor reshaping, FLOP analysis, and configuration.
10. Inference flow
- Inference scripts are dataset-specific, which implies each dataset needs its own label mapping, CSV conventions, or preprocessing logic.
- The flow is: load pretrained VMamba weights → build XFMamba network → load dataset split → run evaluation and metric computation.
- The outputs are likely AUROC and confusion-matrix-style summaries, matching the README’s reported metrics.
11. Artifacts and outputs
- figs/ stores paper figures, not runtime code.
- Training outputs are configured through CLI flags such as --savemodel_path and --confusion_matrix_folder, so checkpoints and evaluation artifacts are externalized rather than hardcoded.
- This means the repo itself is primarily a modeling/training framework, not a packaged library or web service.
12. How to draw the architecture
- A clean diagram would show:
1. Dataset CSV + medical images
2. Dataset-specific two-view loader and preprocessing
3. Shared training/inference engine
4. Dual-branch VMamba backbone
5. Cross-fusion module
6. Classification head
7. Metrics/checkpoint outputs
- Separately, show the compiled Mamba selective-scan extension as a lower-level acceleration layer used by the backbone.
13. Repo shape at a glance
- libs/ = data, training, configuration, utilities
- models/ = backbone, fusion, and Mamba kernels/wrappers
- top-level scripts = experiment entry points
- models/selective_scan/ = native CUDA extension build and tests
Main entry points: 1_train_model.py Train model · net_fusionmamba.py Fusion model · training.py Training loop · nyu_cropping.py Cropping
Components 23
Entry points
Top-level training and dataset-specific inference scripts.
- Train model · Main training script for experiments.
1_train_model.py - Infer MURA · Dataset-specific inference script for MURA.
2_inference_mura.py - Infer CheXpert · Dataset-specific inference script for CheXpert.
2_inference_chexpert.py - Infer DDSM · Dataset-specific inference script for CBIS-DDSM x-ray runs.
2_inference_ddsmxray.py
Data pipeline
Dataset loaders and preprocessing for paired medical views.
- MURA loader · Paired-view dataset reader for MURA.
libs/dataset_mura_twoview.py - CheXpert loader · Paired-view dataset reader for CheXpert.
libs/dataset_chexpert_twoview.py - DDSM loader · Paired-view dataset reader for CBIS-DDSM x-ray.
libs/dataset_ddsmxray_twoview.py - Hip loader · Paired-view dataset reader for hip x-ray experiments.
libs/dataset_hipxray_twoview.py - Cropping · Optional localization and cropping before classification.
libs/nyu_cropping.py - Center calc · Heuristic or learned center estimation for cropping.
libs/calc_optimal_centers.py
Training control
Shared orchestration, configuration, utilities, and stopping logic.
- Training loop · Reusable training engine for optimization and evaluation.
libs/training.py - Mutual learning · Alternative training routine for coupled model learning.
libs/train_mutual_learning.py - Config · Argument and configuration handling.
libs/config.py - Utils · Shared utilities for metrics, checkpoints, and experiment support.
libs/utils.py - Early stop · Reusable early-stopping helper.
libs/early_stop.py - Stop helper · Top-level early stopping helper used by scripts.
early_stop.py
Modeling
Backbone, fusion, and model assembly for cross-view classification.
- Fusion model · Top-level network wiring for dual-view cross-fusion classification.
net_fusionmamba.py - VMamba · Primary visual feature extractor used by the dual-view model.
models/vmamba.py - Cross fusion · Cross-view feature interaction and fusion module.
models/fusion_vmamba.py - Triton ops · Alternative Triton-backed Mamba-related implementation.
models/csm_triton.py - Scan wrapper · Mamba/scan implementation variant used by the model stack.
models/csms6s.py
Acceleration
Low-level Mamba selective-scan implementation and wrappers.
- Mamba2 core · Low-level Mamba2 layers and state update utilities.
models/mamba2 - Selective scan · Compiled selective-scan CUDA/C++ extension used by the backbone.
models/selective_scan
Connections 37
- Train model → Config: parse args · Training script consumes CLI/config settings.
- Train model → MURA loader: build data · Uses dataset-specific loader when training MURA.
- Train model → CheXpert loader: build data · Uses dataset-specific loader when training CheXpert.
- Train model → DDSM loader: build data · Uses dataset-specific loader when training DDSM.
- Train model → Training loop: run train · Hands off to the shared training engine.
- Train model → Fusion model: build model · Constructs the top-level fusion network.
- Infer MURA → Config: parse args · Inference script consumes CLI/config settings.
- Infer MURA → MURA loader: load data · Loads MURA evaluation split.
- Infer MURA → Fusion model: load model · Builds the fusion network for evaluation.
- Infer CheXpert → Config: parse args · Inference script consumes CLI/config settings.
- Infer CheXpert → CheXpert loader: load data · Loads CheXpert evaluation split.
- Infer CheXpert → Fusion model: load model · Builds the fusion network for evaluation.
- Infer DDSM → Config: parse args · Inference script consumes CLI/config settings.
- Infer DDSM → DDSM loader: load data · Loads DDSM evaluation split.
- Infer DDSM → Fusion model: load model · Builds the fusion network for evaluation.
- MURA loader → Cropping: preprocess · Can apply cropping/localization before inference.
- CheXpert loader → Cropping: preprocess · Can apply cropping/localization before inference.
- DDSM loader → Cropping: preprocess · Can apply cropping/localization before inference.
- Hip loader → Cropping: preprocess · Can apply cropping/localization before inference.
- Cropping → Center calc: estimate centers · Optional center estimation supports cropping.
- Training loop → VMamba: optimize · Training updates the VMamba backbone.
- Training loop → Cross fusion: optimize · Training updates the cross-view fusion module.
- Fusion model → VMamba: uses backbone · Top-level model instantiates the feature extractor.
- Fusion model → Cross fusion: uses fusion · Top-level model instantiates cross-view fusion.
- Fusion model → Scan wrapper: uses scan · Model assembly depends on scan-aware Mamba blocks.
- VMamba → Mamba2 core: uses layers · Backbone relies on Mamba2 state-space components.
- Mamba2 core → Selective scan: accelerates · Low-level state-space ops dispatch to CUDA kernels.
- Scan wrapper → Selective scan: accelerates · Scan wrapper uses the compiled selective-scan extension.
- Triton ops → Selective scan: alternative path · Alternative accelerated implementation in the model stack.
- Training loop → Utils: report metrics · Uses shared utility functions for logging and evaluation.
- Training loop → Early stop: stop criteria · Shared training loop consults early-stopping logic.
- Train model → Stop helper: stop criteria · Top-level script can use repository-level early stopping helper.
- Mutual learning → Training loop: variant of · Specialized training routine builds on shared orchestration.
- Mutual learning → Utils: report metrics · Uses shared utility functions for experiment bookkeeping.
- Infer MURA → Utils: metrics · Inference computes and records evaluation metrics.
- Infer CheXpert → Utils: metrics · Inference computes and records evaluation metrics.
- Infer DDSM → Utils: metrics · Inference computes and records evaluation metrics.