# xzheng0427/xfmamba architecture

> Architecture diagram of the GitHub repository xzheng0427/xfmamba, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xzheng0427/xfmamba
- Repository: https://github.com/xzheng0427/xfmamba

## Overview

1. **What this repo is**
   - This is a research training/inference codebase for **multi-view medical image classification**.
   - The core idea is a **cross-fusion Mamba** model that combines features from two views of the same exam, with dataset-specific pipelines for **MURA**, **CheXpert**, and **CBIS-DDSM**.

2. **Top-level execution flow**
   - `1_train_model.py` is the training entry point.
   - `2_inference_mura.py`, `2_inference_chexpert.py`, and `2_inference_ddsmxray.py` are dataset-specific inference entry points.
   - These scripts likely parse CLI arguments, build the dataset/model, then hand control to shared training or evaluation code under `libs/`.

3. **Data boundary**
   - The repository expects dataset metadata in **CSV form** plus image files from the public medical datasets.
   - Dataset-specific loaders live in:
     - `libs/dataset_mura_twoview.py`
     - `libs/dataset_chexpert_twoview.py`
     - `libs/dataset_ddsmxray_twoview.py`
     - `libs/dataset_hipxray_twoview.py`
   - The repeated `twoview` naming is important: the model is not just classifying a single image, but consuming **paired views** as the primary input unit.

4. **Preprocessing / spatial handling**
   - `libs/nyu_cropping.py` and `libs/calc_optimal_centers.py` suggest optional image localization or cropping logic before classification.
   - This preprocessing is part of the data pipeline rather than the model itself, so it sits on the input side of the architecture.

5. **Shared training orchestration**
   - `libs/training.py` and `libs/train_mutual_learning.py` are the main reusable training loops.
   - `libs/early_stop.py` and top-level `early_stop.py` provide stopping logic.
   - `libs/utils.py` and `libs/config.py` likely centralize metrics, argument/config handling, logging, and checkpoint utilities.
   - This is the main “control plane” separating dataset/model-specific code from generic training mechanics.

6. **Model assembly boundary**
   - `net_fusionmamba.py` appears to be the main network definition that wires together the backbone and fusion logic.
   - `models/` contains the model implementations and variants, including:
     - `models/vmamba.py` for the VMamba backbone
     - `models/fusion_vmamba.py` for the cross-view fusion layer
     - `models/csm_triton.py` and `models/csms6s.py` for alternative Mamba/scan implementations or wrappers
   - In the architecture diagram, this is the central compute block between input views and classification head.

7. **Backbone and fusion structure**
   - The project builds on **VMamba** as the feature extractor, then applies **cross-fusion** between the two views.
   - The README’s model name `dualfusionmambav13` hints that the main network is a dual-branch design with feature interaction between branches.
   - The output is a task-specific classifier head with configurable `num_classes`.

8. **Mamba / selective scan subsystem**
   - The `models/mamba2/` directory contains low-level Mamba state-space layers and supporting tensor/state update code.
   - `models/selective_scan/` is a custom **C++/CUDA extension** with:
     - `setup.py`
     - `csrc/` kernels
     - Python bindings in `selective_scan_interface.py`
   - This is a major architectural boundary: part of the computation runs through compiled GPU kernels, not pure Python.

9. **Runtime and external dependencies**
   - The repo is built around **PyTorch 2.2**, **Triton**, **CUDA**, and the external `mamba_ssm` wheel.
   - The selective-scan kernels are performance-critical infrastructure for the Mamba backbone.
   - `timm`, `einops`, `fvcore`, and `yacs` support model building, tensor reshaping, FLOP analysis, and configuration.

10. **Inference flow**
   - Inference scripts are dataset-specific, which implies each dataset needs its own label mapping, CSV conventions, or preprocessing logic.
   - The flow is: load pretrained VMamba weights → build XFMamba network → load dataset split → run evaluation and metric computation.
   - The outputs are likely AUROC and confusion-matrix-style summaries, matching the README’s reported metrics.

11. **Artifacts and outputs**
   - `figs/` stores paper figures, not runtime code.
   - Training outputs are configured through CLI flags such as `--savemodel_path` and `--confusion_matrix_folder`, so checkpoints and evaluation artifacts are externalized rather than hardcoded.
   - This means the repo itself is primarily a **modeling/training framework**, not a packaged library or web service.

12. **How to draw the architecture**
   - A clean diagram would show:
     1. Dataset CSV + medical images
     2. Dataset-specific two-view loader and preprocessing
     3. Shared training/inference engine
     4. Dual-branch VMamba backbone
     5. Cross-fusion module
     6. Classification head
     7. Metrics/checkpoint outputs
   - Separately, show the compiled Mamba selective-scan extension as a lower-level acceleration layer used by the backbone.

13. **Repo shape at a glance**
   - `libs/` = data, training, configuration, utilities
   - `models/` = backbone, fusion, and Mamba kernels/wrappers
   - top-level scripts = experiment entry points
   - `models/selective_scan/` = native CUDA extension build and tests

Main entry points: [`1_train_model.py`](https://github.com/xzheng0427/xfmamba/blob/main/1_train_model.py) (Train model), [`net_fusionmamba.py`](https://github.com/xzheng0427/xfmamba/blob/main/net_fusionmamba.py) (Fusion model), [`libs/training.py`](https://github.com/xzheng0427/xfmamba/blob/main/libs/training.py) (Training loop), [`libs/nyu_cropping.py`](https://github.com/xzheng0427/xfmamba/blob/main/libs/nyu_cropping.py) (Cropping)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entrypoints["Entry points"]
  node_train_entry["Train model<br/>python entrypoint<br/>[1_train_model.py]"]
  node_infer_mura["Infer MURA<br/>python entrypoint"]
  node_infer_chexpert["Infer CheXpert<br/>python entrypoint"]
  node_infer_ddsm["Infer DDSM<br/>python entrypoint"]
end

subgraph group_data["Data pipeline"]
  node_mura_loader["MURA loader<br/>two-view dataset"]
  node_chexpert_loader["CheXpert loader<br/>two-view dataset"]
  node_ddsm_loader["DDSM loader<br/>two-view dataset"]
  node_hip_loader["Hip loader<br/>two-view dataset"]
  node_crop_preprocess["Cropping<br/>image preprocess<br/>[nyu_cropping.py]"]
  node_center_calc["Center calc<br/>preprocess helper"]
end

subgraph group_control["Training control"]
  node_training_loop["Training loop<br/>orchestration<br/>[training.py]"]
  node_mutual_learning["Mutual learning<br/>training variant"]
  node_config["Config<br/>[config.py]"]
  node_utils["Utils<br/>metrics/logging<br/>[utils.py]"]
  node_early_stop_lib["Early stop<br/>stopping logic<br/>[early_stop.py]"]
  node_early_stop_top["Stop helper<br/>stopping logic<br/>[early_stop.py]"]
end

subgraph group_models["Modeling"]
  node_net_assembly["Fusion model<br/>model assembly<br/>[net_fusionmamba.py]"]
  node_vmamba["VMamba<br/>backbone<br/>[vmamba.py]"]
  node_fusion_vmamba["Cross fusion<br/>fusion block<br/>[fusion_vmamba.py]"]
  node_csm_triton["Triton ops<br/>kernel wrapper<br/>[csm_triton.py]"]
  node_csms6s["Scan wrapper<br/>mamba wrapper<br/>[csms6s.py]"]
end

subgraph group_accel["Acceleration"]
  node_mamba2["Mamba2 core<br/>state-space layers"]
  node_selective_scan["Selective scan<br/>cuda extension"]
end

node_train_entry -->|"parse args"| node_config
node_train_entry -->|"build data"| node_mura_loader
node_train_entry -->|"build data"| node_chexpert_loader
node_train_entry -->|"build data"| node_ddsm_loader
node_train_entry -->|"run train"| node_training_loop
node_train_entry -->|"build model"| node_net_assembly
node_infer_mura -->|"parse args"| node_config
node_infer_mura -->|"load data"| node_mura_loader
node_infer_mura -->|"load model"| node_net_assembly
node_infer_chexpert -->|"parse args"| node_config
node_infer_chexpert -->|"load data"| node_chexpert_loader
node_infer_chexpert -->|"load model"| node_net_assembly
node_infer_ddsm -->|"parse args"| node_config
node_infer_ddsm -->|"load data"| node_ddsm_loader
node_infer_ddsm -->|"load model"| node_net_assembly
node_mura_loader -->|"preprocess"| node_crop_preprocess
node_chexpert_loader -->|"preprocess"| node_crop_preprocess
node_ddsm_loader -->|"preprocess"| node_crop_preprocess
node_hip_loader -->|"preprocess"| node_crop_preprocess
node_crop_preprocess -->|"estimate centers"| node_center_calc
node_training_loop -->|"optimize"| node_vmamba
node_training_loop -->|"optimize"| node_fusion_vmamba
node_net_assembly -->|"uses backbone"| node_vmamba
node_net_assembly -->|"uses fusion"| node_fusion_vmamba
node_net_assembly -->|"uses scan"| node_csms6s
node_vmamba -->|"uses layers"| node_mamba2
node_mamba2 -->|"accelerates"| node_selective_scan
node_csms6s -->|"accelerates"| node_selective_scan
node_csm_triton -->|"alternative path"| node_selective_scan
node_training_loop -->|"report metrics"| node_utils
node_training_loop -->|"stop criteria"| node_early_stop_lib
node_train_entry -->|"stop criteria"| node_early_stop_top
node_mutual_learning -->|"variant of"| node_training_loop
node_mutual_learning -->|"report metrics"| node_utils
node_infer_mura -->|"metrics"| node_utils
node_infer_chexpert -->|"metrics"| node_utils
node_infer_ddsm -->|"metrics"| node_utils

click node_train_entry "https://github.com/xzheng0427/xfmamba/blob/main/1_train_model.py"
click node_infer_mura "https://github.com/xzheng0427/xfmamba/blob/main/2_inference_mura.py"
click node_infer_chexpert "https://github.com/xzheng0427/xfmamba/blob/main/2_inference_chexpert.py"
click node_infer_ddsm "https://github.com/xzheng0427/xfmamba/blob/main/2_inference_ddsmxray.py"
click node_net_assembly "https://github.com/xzheng0427/xfmamba/blob/main/net_fusionmamba.py"
click node_vmamba "https://github.com/xzheng0427/xfmamba/blob/main/models/vmamba.py"
click node_fusion_vmamba "https://github.com/xzheng0427/xfmamba/blob/main/models/fusion_vmamba.py"
click node_csm_triton "https://github.com/xzheng0427/xfmamba/blob/main/models/csm_triton.py"
click node_csms6s "https://github.com/xzheng0427/xfmamba/blob/main/models/csms6s.py"
click node_mamba2 "https://github.com/xzheng0427/xfmamba/tree/main/models/mamba2"
click node_selective_scan "https://github.com/xzheng0427/xfmamba/tree/main/models/selective_scan"
click node_mura_loader "https://github.com/xzheng0427/xfmamba/blob/main/libs/dataset_mura_twoview.py"
click node_chexpert_loader "https://github.com/xzheng0427/xfmamba/blob/main/libs/dataset_chexpert_twoview.py"
click node_ddsm_loader "https://github.com/xzheng0427/xfmamba/blob/main/libs/dataset_ddsmxray_twoview.py"
click node_hip_loader "https://github.com/xzheng0427/xfmamba/blob/main/libs/dataset_hipxray_twoview.py"
click node_crop_preprocess "https://github.com/xzheng0427/xfmamba/blob/main/libs/nyu_cropping.py"
click node_center_calc "https://github.com/xzheng0427/xfmamba/blob/main/libs/calc_optimal_centers.py"
click node_training_loop "https://github.com/xzheng0427/xfmamba/blob/main/libs/training.py"
click node_mutual_learning "https://github.com/xzheng0427/xfmamba/blob/main/libs/train_mutual_learning.py"
click node_config "https://github.com/xzheng0427/xfmamba/blob/main/libs/config.py"
click node_utils "https://github.com/xzheng0427/xfmamba/blob/main/libs/utils.py"
click node_early_stop_lib "https://github.com/xzheng0427/xfmamba/blob/main/libs/early_stop.py"
click node_early_stop_top "https://github.com/xzheng0427/xfmamba/blob/main/early_stop.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_train_entry,node_infer_mura,node_infer_chexpert,node_infer_ddsm toneBlue
class node_mura_loader,node_chexpert_loader,node_ddsm_loader,node_hip_loader,node_crop_preprocess,node_center_calc toneAmber
class node_training_loop,node_mutual_learning,node_config,node_utils,node_early_stop_lib,node_early_stop_top toneMint
class node_net_assembly,node_vmamba,node_fusion_vmamba,node_csm_triton,node_csms6s toneRose
class node_mamba2,node_selective_scan toneIndigo
```

## Components

### Entry points

Top-level training and dataset-specific inference scripts.

- [Train model](https://github.com/xzheng0427/xfmamba/blob/main/1_train_model.py): python entrypoint · Main training script for experiments. · `1_train_model.py`
- [Infer MURA](https://github.com/xzheng0427/xfmamba/blob/main/2_inference_mura.py): python entrypoint · Dataset-specific inference script for MURA. · `2_inference_mura.py`
- [Infer CheXpert](https://github.com/xzheng0427/xfmamba/blob/main/2_inference_chexpert.py): python entrypoint · Dataset-specific inference script for CheXpert. · `2_inference_chexpert.py`
- [Infer DDSM](https://github.com/xzheng0427/xfmamba/blob/main/2_inference_ddsmxray.py): python entrypoint · Dataset-specific inference script for CBIS-DDSM x-ray runs. · `2_inference_ddsmxray.py`

### Data pipeline

Dataset loaders and preprocessing for paired medical views.

- [MURA loader](https://github.com/xzheng0427/xfmamba/blob/main/libs/dataset_mura_twoview.py): two-view dataset · Paired-view dataset reader for MURA. · `libs/dataset_mura_twoview.py`
- [CheXpert loader](https://github.com/xzheng0427/xfmamba/blob/main/libs/dataset_chexpert_twoview.py): two-view dataset · Paired-view dataset reader for CheXpert. · `libs/dataset_chexpert_twoview.py`
- [DDSM loader](https://github.com/xzheng0427/xfmamba/blob/main/libs/dataset_ddsmxray_twoview.py): two-view dataset · Paired-view dataset reader for CBIS-DDSM x-ray. · `libs/dataset_ddsmxray_twoview.py`
- [Hip loader](https://github.com/xzheng0427/xfmamba/blob/main/libs/dataset_hipxray_twoview.py): two-view dataset · Paired-view dataset reader for hip x-ray experiments. · `libs/dataset_hipxray_twoview.py`
- [Cropping](https://github.com/xzheng0427/xfmamba/blob/main/libs/nyu_cropping.py): image preprocess · Optional localization and cropping before classification. · `libs/nyu_cropping.py`
- [Center calc](https://github.com/xzheng0427/xfmamba/blob/main/libs/calc_optimal_centers.py): preprocess helper · Heuristic or learned center estimation for cropping. · `libs/calc_optimal_centers.py`

### Training control

Shared orchestration, configuration, utilities, and stopping logic.

- [Training loop](https://github.com/xzheng0427/xfmamba/blob/main/libs/training.py): orchestration · Reusable training engine for optimization and evaluation. · `libs/training.py`
- [Mutual learning](https://github.com/xzheng0427/xfmamba/blob/main/libs/train_mutual_learning.py): training variant · Alternative training routine for coupled model learning. · `libs/train_mutual_learning.py`
- [Config](https://github.com/xzheng0427/xfmamba/blob/main/libs/config.py): runtime config · Argument and configuration handling. · `libs/config.py`
- [Utils](https://github.com/xzheng0427/xfmamba/blob/main/libs/utils.py): metrics/logging · Shared utilities for metrics, checkpoints, and experiment support. · `libs/utils.py`
- [Early stop](https://github.com/xzheng0427/xfmamba/blob/main/libs/early_stop.py): stopping logic · Reusable early-stopping helper. · `libs/early_stop.py`
- [Stop helper](https://github.com/xzheng0427/xfmamba/blob/main/early_stop.py): stopping logic · Top-level early stopping helper used by scripts. · `early_stop.py`

### Modeling

Backbone, fusion, and model assembly for cross-view classification.

- [Fusion model](https://github.com/xzheng0427/xfmamba/blob/main/net_fusionmamba.py): model assembly · Top-level network wiring for dual-view cross-fusion classification. · `net_fusionmamba.py`
- [VMamba](https://github.com/xzheng0427/xfmamba/blob/main/models/vmamba.py): backbone · Primary visual feature extractor used by the dual-view model. · `models/vmamba.py`
- [Cross fusion](https://github.com/xzheng0427/xfmamba/blob/main/models/fusion_vmamba.py): fusion block · Cross-view feature interaction and fusion module. · `models/fusion_vmamba.py`
- [Triton ops](https://github.com/xzheng0427/xfmamba/blob/main/models/csm_triton.py): kernel wrapper · Alternative Triton-backed Mamba-related implementation. · `models/csm_triton.py`
- [Scan wrapper](https://github.com/xzheng0427/xfmamba/blob/main/models/csms6s.py): mamba wrapper · Mamba/scan implementation variant used by the model stack. · `models/csms6s.py`

### Acceleration

Low-level Mamba selective-scan implementation and wrappers.

- [Mamba2 core](https://github.com/xzheng0427/xfmamba/tree/main/models/mamba2): state-space layers · Low-level Mamba2 layers and state update utilities. · `models/mamba2`
- [Selective scan](https://github.com/xzheng0427/xfmamba/tree/main/models/selective_scan): cuda extension · Compiled selective-scan CUDA/C++ extension used by the backbone. · `models/selective_scan`

## Connections

- Train model → Config: parse args · Training script consumes CLI/config settings.
- Train model → MURA loader: build data · Uses dataset-specific loader when training MURA.
- Train model → CheXpert loader: build data · Uses dataset-specific loader when training CheXpert.
- Train model → DDSM loader: build data · Uses dataset-specific loader when training DDSM.
- Train model → Training loop: run train · Hands off to the shared training engine.
- Train model → Fusion model: build model · Constructs the top-level fusion network.
- Infer MURA → Config: parse args · Inference script consumes CLI/config settings.
- Infer MURA → MURA loader: load data · Loads MURA evaluation split.
- Infer MURA → Fusion model: load model · Builds the fusion network for evaluation.
- Infer CheXpert → Config: parse args · Inference script consumes CLI/config settings.
- Infer CheXpert → CheXpert loader: load data · Loads CheXpert evaluation split.
- Infer CheXpert → Fusion model: load model · Builds the fusion network for evaluation.
- Infer DDSM → Config: parse args · Inference script consumes CLI/config settings.
- Infer DDSM → DDSM loader: load data · Loads DDSM evaluation split.
- Infer DDSM → Fusion model: load model · Builds the fusion network for evaluation.
- MURA loader → Cropping: preprocess · Can apply cropping/localization before inference.
- CheXpert loader → Cropping: preprocess · Can apply cropping/localization before inference.
- DDSM loader → Cropping: preprocess · Can apply cropping/localization before inference.
- Hip loader → Cropping: preprocess · Can apply cropping/localization before inference.
- Cropping → Center calc: estimate centers · Optional center estimation supports cropping.
- Training loop → VMamba: optimize · Training updates the VMamba backbone.
- Training loop → Cross fusion: optimize · Training updates the cross-view fusion module.
- Fusion model → VMamba: uses backbone · Top-level model instantiates the feature extractor.
- Fusion model → Cross fusion: uses fusion · Top-level model instantiates cross-view fusion.
- Fusion model → Scan wrapper: uses scan · Model assembly depends on scan-aware Mamba blocks.
- VMamba → Mamba2 core: uses layers · Backbone relies on Mamba2 state-space components.
- Mamba2 core → Selective scan: accelerates · Low-level state-space ops dispatch to CUDA kernels.
- Scan wrapper → Selective scan: accelerates · Scan wrapper uses the compiled selective-scan extension.
- Triton ops → Selective scan: alternative path · Alternative accelerated implementation in the model stack.
- Training loop → Utils: report metrics · Uses shared utility functions for logging and evaluation.
- Training loop → Early stop: stop criteria · Shared training loop consults early-stopping logic.
- Train model → Stop helper: stop criteria · Top-level script can use repository-level early stopping helper.
- Mutual learning → Training loop: variant of · Specialized training routine builds on shared orchestration.
- Mutual learning → Utils: report metrics · Uses shared utility functions for experiment bookkeeping.
- Infer MURA → Utils: metrics · Inference computes and records evaluation metrics.
- Infer CheXpert → Utils: metrics · Inference computes and records evaluation metrics.
- Infer DDSM → Utils: metrics · Inference computes and records evaluation metrics.
