# alibaba-damo-academy/clinfusion architecture

> Architecture diagram of the GitHub repository alibaba-damo-academy/clinfusion, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/alibaba-damo-academy/clinfusion
- Repository: https://github.com/alibaba-damo-academy/clinfusion

## Overview

#### Purpose and Entry Points
ClinFusion is a Python medical multimodal LLM package for text, 2D medical images, and 3D NIfTI volumes. The operational entry points are inference/evaluation launch scripts, which drive the evaluation coordinator at `eval/Evaluation/main.py` and generation rollout at `eval/InferenceEngine/rollout.py`. Inputs and outputs are JSONL, preserving source fields and adding model generations.

#### Multimodal Model Assembly
The ClinFusion-specific model implementation adapts Qwen VL architectures with multiple medical vision encoders:
- Qwen3-VL multi-encoder model: `custom_model/modeling_multi_enc_qwen3_vl.py`
- Qwen2.5-VL variant: `custom_model/modeling_multi_enc_qwen2_5_vl.py`
- model loading/registration: `custom_model/loader.py`
- shared encoder fusion utilities: `custom_model/multi_enc_utils.py`

The repository expects externally downloaded Qwen base-model and ClinFusion checkpoint weights rather than containing weights itself.

#### Vision Encoding and Medical Input Processing
The visual pipeline combines external DINOv2 and OpenCLIP/ConvNeXt encoders with an in-repo medical ViT. `custom_model/openclip_encoder.py` wraps the OpenCLIP encoder; the medical ViT is constructed through `custom_model/medical_vit/create_medical_vit.py` and implemented in `custom_model/medical_vit/mae_v2.py`. Image/volume transformation and multimodal processor behavior are defined in `custom_model/medical_image_preprocessing.py` and `custom_model/image_processing_multi_enc_qwen2_vl.py`.

A key modality boundary is NIfTI-to-2D-slice conversion before model processing, allowing 3D CT/MRI inputs to enter the vision-language generation path.

#### Inference Engine and Model Abstraction
`eval/InferenceEngine/rollout.py` orchestrates distributed/local generation over JSONL records. Model selection is isolated behind wrapper resolution in `eval/InferenceEngine/models/get_wrapper.py`. Supported routes include ClinFusion/custom local checkpoints (`eval/InferenceEngine/models/custom.py`), Qwen VL (`eval/InferenceEngine/models/qwenvl.py`), Hulu-Med, and API-backed models (`eval/InferenceEngine/models/api.py`).

Custom checkpoints implement the adapter contract in `custom_model/medevalkit_adapter_qwen3_vl.py`; this separates evaluation infrastructure from model-specific loading, preprocessing, and generation APIs.

#### Evaluation Pipeline
`eval/Evaluation/main.py` loads a YAML-selected dataset, model wrapper, evaluator, and aggregator. Evaluator branches represent major task classes:
- general 2D/text tasks: `eval/Evaluation/evaluators/evaluator_general.py`
- volumetric tasks: `eval/Evaluation/evaluators/evaluator_3d.py`
- instruction-following tasks: `eval/Evaluation/evaluators/evaluator_if.py`

Their outputs feed pluggable metric/report aggregation in `eval/Evaluation/aggregators/aggregator.py`, `eval/Evaluation/aggregators/aggregator_if.py`, and `eval/Evaluation/aggregators/aggregator_staging.py`.

#### Instruction-Following Tagging Boundary
Instruction-following evaluation has a dedicated prompt/tag pipeline: `eval/tagging/prepare_prompts.py` prepares tagged prompts, while `eval/tagging/parse_and_validate_tags.py` parses and validates generated tags. This separates structured compliance/factualness assessment from normal generation and task scoring.

#### Runtime, Infrastructure, and External Services
The primary runtime is Python 3.11 with PyTorch/CUDA and a prebuilt Flash-Attention wheel, provisioned by `install_from_scratch.sh` and `requirements.txt`. Local execution supports single- and multi-node GPU workers through launch scripts such as `eval/tools/launch_inference.sh` and `eval/tools/launch_eval.sh`. External dependencies include Hugging Face model hosting/downloads, DINOv2, OpenCLIP ConvNeXt, Qwen VL model weights, and configurable API model endpoints/credentials via `eval/InferenceEngine/models/api.py`.

Main entry points: [`eval/InferenceEngine/rollout.py`](https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/InferenceEngine/rollout.py) (Generation rollout), [`eval/InferenceEngine/models/get_wrapper.py`](https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/InferenceEngine/models/get_wrapper.py) (Wrapper resolution), [`custom_model/medevalkit_adapter_qwen3_vl.py`](https://github.com/alibaba-damo-academy/clinfusion/blob/master/custom_model/medevalkit_adapter_qwen3_vl.py) (ClinFusion adapter), [`eval/Evaluation/main.py`](https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/Evaluation/main.py) (Evaluation coordinator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime & Entry Points"]
  node_launch["GPU launch scripts<br/>shell entry points"]
end

subgraph group_evaluation["Evaluation System"]
  node_eval_main["Evaluation coordinator<br/>Python coordinator<br/>[main.py]"]
  node_evaluators["Task evaluators<br/>evaluation branches"]
  node_aggregators["Metrics &amp; reports<br/>aggregators<br/>[aggregator.py]"]
  node_tag_prompts["Tagged prompt preparation<br/>tagging pipeline<br/>[prepare_prompts.py]"]
  node_tag_validation["Tag parsing &amp; validation<br/>tagging pipeline"]
end

subgraph group_inference["Inference Engine"]
  node_rollout["Generation rollout<br/>JSONL generation orchestrator<br/>[rollout.py]"]
  node_wrapper_resolution["Wrapper resolution<br/>model router<br/>[get_wrapper.py]"]
  node_custom_wrapper["Custom checkpoint wrapper<br/>local model wrapper<br/>[custom.py]"]
  node_api_wrapper["API model wrapper<br/>remote model wrapper<br/>[api.py]"]
end

subgraph group_model["ClinFusion Model"]
  node_adapter["ClinFusion adapter<br/>adapter contract"]
  node_model_loader["Model loader &amp; registration<br/>model assembly<br/>[loader.py]"]
  node_qwen_model["Multi-encoder Qwen VL<br/>vision-language model"]
  node_fusion["Encoder fusion utilities<br/>multimodal fusion<br/>[multi_enc_utils.py]"]
end

subgraph group_vision["Medical Vision Pipeline"]
  node_medical_processing["Medical input processor<br/>image and volume processing"]
  node_multienc_processor["Multi-encoder processor<br/>multimodal processor"]
  node_openclip["OpenCLIP / ConvNeXt encoder<br/>external vision encoder wrapper"]
  node_medical_vit["Medical ViT<br/>in-repo vision encoder"]
end

node_external_weights[("External models &amp; weights<br/>external dependencies")]
node_jsonl["JSONL datasets &amp; generations<br/>data artifacts"]

node_launch -->|"launches"| node_eval_main
node_launch -->|"launches"| node_rollout
node_eval_main -->|"selects model"| node_wrapper_resolution
node_eval_main -->|"dispatches tasks"| node_evaluators
node_evaluators -->|"scores feed"| node_aggregators
node_evaluators -->|"instruction tasks"| node_tag_prompts
node_tag_prompts -->|"tagged requests"| node_rollout
node_rollout -->|"generated tags"| node_tag_validation
node_tag_validation -->|"validated assessment"| node_aggregators
node_jsonl -->|"input records"| node_rollout
node_rollout -->|"augmented outputs"| node_jsonl
node_rollout -->|"resolves model"| node_wrapper_resolution
node_wrapper_resolution -->|"local ClinFusion route"| node_custom_wrapper
node_wrapper_resolution -.->|"API route"| node_api_wrapper
node_custom_wrapper -->|"uses"| node_adapter
node_adapter -->|"loads"| node_model_loader
node_model_loader -->|"registers"| node_qwen_model
node_medical_processing -->|"prepared slices and images"| node_multienc_processor
node_multienc_processor -->|"multimodal inputs"| node_adapter
node_adapter -->|"preprocesses"| node_medical_processing
node_qwen_model -->|"fuses encoders"| node_fusion
node_fusion -->|"uses features"| node_openclip
node_fusion -->|"uses features"| node_medical_vit
node_external_weights -.->|"base and checkpoint weights"| node_model_loader
node_external_weights -.->|"encoder weights"| node_openclip

click node_launch "https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/tools/launch_inference.sh"
click node_eval_main "https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/Evaluation/main.py"
click node_evaluators "https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/Evaluation/evaluators/evaluator_general.py"
click node_aggregators "https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/Evaluation/aggregators/aggregator.py"
click node_tag_prompts "https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/tagging/prepare_prompts.py"
click node_tag_validation "https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/tagging/parse_and_validate_tags.py"
click node_rollout "https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/InferenceEngine/rollout.py"
click node_wrapper_resolution "https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/InferenceEngine/models/get_wrapper.py"
click node_custom_wrapper "https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/InferenceEngine/models/custom.py"
click node_api_wrapper "https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/InferenceEngine/models/api.py"
click node_adapter "https://github.com/alibaba-damo-academy/clinfusion/blob/master/custom_model/medevalkit_adapter_qwen3_vl.py"
click node_model_loader "https://github.com/alibaba-damo-academy/clinfusion/blob/master/custom_model/loader.py"
click node_qwen_model "https://github.com/alibaba-damo-academy/clinfusion/blob/master/custom_model/modeling_multi_enc_qwen3_vl.py"
click node_fusion "https://github.com/alibaba-damo-academy/clinfusion/blob/master/custom_model/multi_enc_utils.py"
click node_medical_processing "https://github.com/alibaba-damo-academy/clinfusion/blob/master/custom_model/medical_image_preprocessing.py"
click node_multienc_processor "https://github.com/alibaba-damo-academy/clinfusion/blob/master/custom_model/image_processing_multi_enc_qwen2_vl.py"
click node_openclip "https://github.com/alibaba-damo-academy/clinfusion/blob/master/custom_model/openclip_encoder.py"
click node_medical_vit "https://github.com/alibaba-damo-academy/clinfusion/blob/master/custom_model/medical_vit/create_medical_vit.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_launch toneBlue
class node_eval_main,node_evaluators,node_aggregators,node_tag_prompts,node_tag_validation toneAmber
class node_rollout,node_wrapper_resolution,node_custom_wrapper,node_api_wrapper toneMint
class node_adapter,node_model_loader,node_qwen_model,node_fusion toneRose
class node_medical_processing,node_multienc_processor,node_openclip,node_medical_vit toneIndigo
class node_external_weights,node_jsonl toneNeutral
```

## Components

### Runtime & Entry Points

- [GPU launch scripts](https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/tools/launch_inference.sh): shell entry points · Launch local or distributed inference and evaluation. · `eval/tools/launch_inference.sh`

### Evaluation System

- [Evaluation coordinator](https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/Evaluation/main.py): Python coordinator · Selects datasets, wrappers, evaluators, and aggregators from YAML. · `eval/Evaluation/main.py`
- [Task evaluators](https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/Evaluation/evaluators/evaluator_general.py): evaluation branches · General, volumetric, and instruction-following task evaluation. · `eval/Evaluation/evaluators/evaluator_general.py`
- [Metrics & reports](https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/Evaluation/aggregators/aggregator.py): aggregators · Produces task-specific metric and report outputs. · `eval/Evaluation/aggregators/aggregator.py`
- [Tagged prompt preparation](https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/tagging/prepare_prompts.py): tagging pipeline · `eval/tagging/prepare_prompts.py`
- [Tag parsing & validation](https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/tagging/parse_and_validate_tags.py): tagging pipeline · Separates structured compliance and factualness assessment. · `eval/tagging/parse_and_validate_tags.py`

### Inference Engine

- [Generation rollout](https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/InferenceEngine/rollout.py): JSONL generation orchestrator · Runs local or distributed generation while preserving source records. · `eval/InferenceEngine/rollout.py`
- [Wrapper resolution](https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/InferenceEngine/models/get_wrapper.py): model router · Selects custom, Qwen VL, Hulu-Med, or API model routes. · `eval/InferenceEngine/models/get_wrapper.py`
- [Custom checkpoint wrapper](https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/InferenceEngine/models/custom.py): local model wrapper · Bridges evaluation infrastructure to ClinFusion loading and generation. · `eval/InferenceEngine/models/custom.py`
- [API model wrapper](https://github.com/alibaba-damo-academy/clinfusion/blob/master/eval/InferenceEngine/models/api.py): remote model wrapper · Uses configured endpoint credentials for API-backed models. · `eval/InferenceEngine/models/api.py`

### ClinFusion Model

- [ClinFusion adapter](https://github.com/alibaba-damo-academy/clinfusion/blob/master/custom_model/medevalkit_adapter_qwen3_vl.py): adapter contract · Provides model-specific loading, preprocessing, and generation APIs. · `custom_model/medevalkit_adapter_qwen3_vl.py`
- [Model loader & registration](https://github.com/alibaba-damo-academy/clinfusion/blob/master/custom_model/loader.py): model assembly · `custom_model/loader.py`
- [Multi-encoder Qwen VL](https://github.com/alibaba-damo-academy/clinfusion/blob/master/custom_model/modeling_multi_enc_qwen3_vl.py): vision-language model · ClinFusion adapts Qwen VL with fused medical vision encoders. · `custom_model/modeling_multi_enc_qwen3_vl.py`
- [Encoder fusion utilities](https://github.com/alibaba-damo-academy/clinfusion/blob/master/custom_model/multi_enc_utils.py): multimodal fusion · `custom_model/multi_enc_utils.py`

### Medical Vision Pipeline

- [Medical input processor](https://github.com/alibaba-damo-academy/clinfusion/blob/master/custom_model/medical_image_preprocessing.py): image and volume processing · Transforms images and converts NIfTI volumes into 2D slices. · `custom_model/medical_image_preprocessing.py`
- [Multi-encoder processor](https://github.com/alibaba-damo-academy/clinfusion/blob/master/custom_model/image_processing_multi_enc_qwen2_vl.py): multimodal processor · `custom_model/image_processing_multi_enc_qwen2_vl.py`
- [OpenCLIP / ConvNeXt encoder](https://github.com/alibaba-damo-academy/clinfusion/blob/master/custom_model/openclip_encoder.py): external vision encoder wrapper · `custom_model/openclip_encoder.py`
- [Medical ViT](https://github.com/alibaba-damo-academy/clinfusion/blob/master/custom_model/medical_vit/create_medical_vit.py): in-repo vision encoder · Constructed medical image encoder used alongside external encoders. · `custom_model/medical_vit/create_medical_vit.py`

### Other components

- **External models & weights**: external dependencies · Hugging Face Qwen and ClinFusion checkpoints plus DINOv2 and OpenCLIP weights.
- **JSONL datasets & generations**: data artifacts · Input records are preserved and augmented with generations.

## Connections

- GPU launch scripts → Evaluation coordinator: launches
- GPU launch scripts → Generation rollout: launches
- Evaluation coordinator → Wrapper resolution: selects model
- Evaluation coordinator → Task evaluators: dispatches tasks
- Task evaluators → Metrics & reports: scores feed
- Task evaluators → Tagged prompt preparation: instruction tasks
- Tagged prompt preparation → Generation rollout: tagged requests
- Generation rollout → Tag parsing & validation: generated tags
- Tag parsing & validation → Metrics & reports: validated assessment
- JSONL datasets & generations → Generation rollout: input records
- Generation rollout → JSONL datasets & generations: augmented outputs
- Generation rollout → Wrapper resolution: resolves model
- Wrapper resolution → Custom checkpoint wrapper: local ClinFusion route
- Wrapper resolution → API model wrapper: API route
- Custom checkpoint wrapper → ClinFusion adapter: uses
- ClinFusion adapter → Model loader & registration: loads
- Model loader & registration → Multi-encoder Qwen VL: registers
- Medical input processor → Multi-encoder processor: prepared slices and images
- Multi-encoder processor → ClinFusion adapter: multimodal inputs
- ClinFusion adapter → Medical input processor: preprocesses
- Multi-encoder Qwen VL → Encoder fusion utilities: fuses encoders
- Encoder fusion utilities → OpenCLIP / ConvNeXt encoder: uses features
- Encoder fusion utilities → Medical ViT: uses features
- External models & weights → Model loader & registration: base and checkpoint weights
- External models & weights → OpenCLIP / ConvNeXt encoder: encoder weights