alibaba-damo-academy/clinfusion

Purpose and Entry Points

ClinFusion is a Python medical multimodal LLM package for text, 2D medical images, and 3D NIfTI volumes. The operational entry points are inference/evaluation launch scripts, which drive the evaluation coordinator at eval/Evaluation/main.py and generation rollout at eval/InferenceEngine/rollout.py. Inputs and outputs are JSONL, preserving source fields and adding model generations.

Multimodal Model Assembly

The ClinFusion-specific model implementation adapts Qwen VL architectures with multiple medical vision encoders:

• Qwen3-VL multi-encoder model: custom_model/modeling_multi_enc_qwen3_vl.py

• Qwen2.5-VL variant: custom_model/modeling_multi_enc_qwen2_5_vl.py

• model loading/registration: custom_model/loader.py

• shared encoder fusion utilities: custom_model/multi_enc_utils.py

The repository expects externally downloaded Qwen base-model and ClinFusion checkpoint weights rather than containing weights itself.

Vision Encoding and Medical Input Processing

The visual pipeline combines external DINOv2 and OpenCLIP/ConvNeXt encoders with an in-repo medical ViT. custom_model/openclip_encoder.py wraps the OpenCLIP encoder; the medical ViT is constructed through custom_model/medical_vit/create_medical_vit.py and implemented in custom_model/medical_vit/mae_v2.py. Image/volume transformation and multimodal processor behavior are defined in custom_model/medical_image_preprocessing.py and custom_model/image_processing_multi_enc_qwen2_vl.py.

A key modality boundary is NIfTI-to-2D-slice conversion before model processing, allowing 3D CT/MRI inputs to enter the vision-language generation path.

Inference Engine and Model Abstraction

eval/InferenceEngine/rollout.py orchestrates distributed/local generation over JSONL records. Model selection is isolated behind wrapper resolution in eval/InferenceEngine/models/get_wrapper.py. Supported routes include ClinFusion/custom local checkpoints (eval/InferenceEngine/models/custom.py), Qwen VL (eval/InferenceEngine/models/qwenvl.py), Hulu-Med, and API-backed models (eval/InferenceEngine/models/api.py).

Custom checkpoints implement the adapter contract in custom_model/medevalkit_adapter_qwen3_vl.py; this separates evaluation infrastructure from model-specific loading, preprocessing, and generation APIs.

Evaluation Pipeline

eval/Evaluation/main.py loads a YAML-selected dataset, model wrapper, evaluator, and aggregator. Evaluator branches represent major task classes:

• general 2D/text tasks: eval/Evaluation/evaluators/evaluator_general.py

• volumetric tasks: eval/Evaluation/evaluators/evaluator_3d.py

• instruction-following tasks: eval/Evaluation/evaluators/evaluator_if.py

Their outputs feed pluggable metric/report aggregation in eval/Evaluation/aggregators/aggregator.py, eval/Evaluation/aggregators/aggregator_if.py, and eval/Evaluation/aggregators/aggregator_staging.py.

Instruction-Following Tagging Boundary

Instruction-following evaluation has a dedicated prompt/tag pipeline: eval/tagging/prepare_prompts.py prepares tagged prompts, while eval/tagging/parse_and_validate_tags.py parses and validates generated tags. This separates structured compliance/factualness assessment from normal generation and task scoring.

Runtime, Infrastructure, and External Services

The primary runtime is Python 3.11 with PyTorch/CUDA and a prebuilt Flash-Attention wheel, provisioned by install_from_scratch.sh and requirements.txt. Local execution supports single- and multi-node GPU workers through launch scripts such as eval/tools/launch_inference.sh and eval/tools/launch_eval.sh. External dependencies include Hugging Face model hosting/downloads, DINOv2, OpenCLIP ConvNeXt, Qwen VL model weights, and configurable API model endpoints/credentials via eval/InferenceEngine/models/api.py.