How mic-dkfz/multitalent works
Project Type & Purpose
• MultiTalent is a Python-based, command-line-driven biomedical image segmentation framework built on top of nnU-Net v2.
• It orchestrates the entire pipeline: dataset conversion → experiment planning → preprocessing → model training → inference → postprocessing → evaluation → optional ensembling or model sharing.
Top-Level Components
A. CLI / Entry Points
• setup.py / pyproject.toml installs console scripts:
– prepare_MT_training (dataset preparation)
– multitalent_train (training)
– multitalent_predict_from_modelfolder (inference)
• These call into multitalent/run/* and multitalent/model_sharing/entry_points.py.
B. Dataset Conversion
• multitalent/dataset_conversion/*.py
• Converts arbitrary 3D datasets into nnU-Net format and generates dataset.json.
C. Experiment Planning
• multitalent/experiment_planning/*
• Extracts dataset fingerprints → computes preprocessing plans → writes out plans for training.
D. Preprocessing
• multitalent/preprocessing/*
• Cropping, normalization, resampling modules plugged into pipelines defined by experiment plans.
E. Model Architectures
• multitalent/architectures/* and multitalent/network_architecture/*
• Host network definitions (e.g., ResEncUNet, MedNeXt, HRNet) that trainers will use.
F. Training
• multitalent/training/* + multitalent/run/run_training*.py
• Data loaders, augmentations, losses, optimizers, LR schedulers, trainer classes (nnUNetTrainer variants).
G. Inference
• multitalent/inference/*
• Sliding-window inference, export predictions, handle multi-dataset segmentation heads.
H. Postprocessing
• multitalent/postprocessing/*
• Optional cleanup of connected components, region cropping.
I. Evaluation & Ensembling
• multitalent/evaluation/* and multitalent/ensembling/*
• Metrics computation, cross-validation summarization, ensemble voting.
J. Batch Running / Benchmarking
• multitalent/batch_running/*
• Generates and submits LSF jobs, collects results across experiments or HPC runs.
K. Model Sharing
• multitalent/model_sharing/*
• Export/import pretrained MultiTalent models, download from remote.
L. Utilities & I/O Registry
• multitalent/utilities/*, multitalent/imageio/*
• File utilities, data loader helpers, reader/writer registry for different image formats.
Core Relationships & Data Flow
• User issues a CLI command → Entry Point orchestrator
→ DatasetConversion (if needed)
→ ExperimentPlanning → outputs plan JSONs
→ Preprocessing pipelines apply to raw data → produce preprocessed training data
→ Training module consumes plans + data → outputs trained model folders (with folds)
→ Inference module reads model + raw/normalized data → outputs segmentation predictions
→ Postprocessing tweaks predictions → stored on disk
→ Evaluation reads predictions + ground truth → calculates metrics → stored as CSV/reports
→ (Optional) Ensembling merges multiple fold predictions → final result
→ (Optional) ModelSharing pushes or pulls model artifacts.
• BatchRunning wraps any of the above in HPC job submission → collects results centrally.
Architectural Patterns & Design Principles
• Layered pipeline: input → transformation → model → output
• Plugin/registry pattern for:
• Architectures (dynamic import by name)
• Reader/writer formats in imageio
• Trainer variants (via nnUNetTrainer subclasses)
• Clear separation of concerns: conversion, planning, preprocessing, training, inference, evaluation.
• Extensive use of JSON/YAML “plans” for configuration, enabling reproducibility.
Technologies & Dependencies
• Python 3.x, PyTorch
• nnU-Net v2 (as a library dependency)
• nibabel, SimpleITK, tifffile for I/O
• HPC scheduler interface (LSF scripts in batch_running)
• Standard Python CLI via setuptools entry points