# mic-dkfz/multitalent architecture

> Architecture diagram of the GitHub repository mic-dkfz/multitalent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mic-dkfz/multitalent
- Repository: https://github.com/mic-dkfz/multitalent

## Overview

##### Project Type & Purpose
- MultiTalent is a Python-based, command-line-driven biomedical image segmentation framework built on top of nnU-Net v2.
- It orchestrates the entire pipeline: dataset conversion → experiment planning → preprocessing → model training → inference → postprocessing → evaluation → optional ensembling or model sharing.

##### Top-Level Components
A. CLI / Entry Points
- `setup.py` / `pyproject.toml` installs console scripts:
– `prepare_MT_training` (dataset preparation)
– `multitalent_train` (training)
– `multitalent_predict_from_modelfolder` (inference)
- These call into `multitalent/run/*` and `multitalent/model_sharing/entry_points.py`.
B. Dataset Conversion
- `multitalent/dataset_conversion/*.py`
- Converts arbitrary 3D datasets into nnU-Net format and generates `dataset.json`.
C. Experiment Planning
- `multitalent/experiment_planning/*`
- Extracts dataset fingerprints → computes preprocessing plans → writes out plans for training.
D. Preprocessing
- `multitalent/preprocessing/*`
- Cropping, normalization, resampling modules plugged into pipelines defined by experiment plans.
E. Model Architectures
- `multitalent/architectures/*` and `multitalent/network_architecture/*`
- Host network definitions (e.g., ResEncUNet, MedNeXt, HRNet) that trainers will use.
F. Training
- `multitalent/training/*` + `multitalent/run/run_training*.py`
- Data loaders, augmentations, losses, optimizers, LR schedulers, trainer classes (nnUNetTrainer variants).
G. Inference
- `multitalent/inference/*`
- Sliding-window inference, export predictions, handle multi-dataset segmentation heads.
H. Postprocessing
- `multitalent/postprocessing/*`
- Optional cleanup of connected components, region cropping.
I. Evaluation & Ensembling
- `multitalent/evaluation/*` and `multitalent/ensembling/*`
- Metrics computation, cross-validation summarization, ensemble voting.
J. Batch Running / Benchmarking
- `multitalent/batch_running/*`
- Generates and submits LSF jobs, collects results across experiments or HPC runs.
K. Model Sharing
- `multitalent/model_sharing/*`
- Export/import pretrained MultiTalent models, download from remote.
L. Utilities & I/O Registry
- `multitalent/utilities/*`, `multitalent/imageio/*`
- File utilities, data loader helpers, reader/writer registry for different image formats.

##### Core Relationships & Data Flow
- User issues a CLI command → Entry Point orchestrator
→ DatasetConversion (if needed)
→ ExperimentPlanning → outputs plan JSONs
→ Preprocessing pipelines apply to raw data → produce preprocessed training data
→ Training module consumes plans + data → outputs trained model folders (with folds)
→ Inference module reads model + raw/normalized data → outputs segmentation predictions
→ Postprocessing tweaks predictions → stored on disk
→ Evaluation reads predictions + ground truth → calculates metrics → stored as CSV/reports
→ (Optional) Ensembling merges multiple fold predictions → final result
→ (Optional) ModelSharing pushes or pulls model artifacts.
- BatchRunning wraps any of the above in HPC job submission → collects results centrally.

##### Architectural Patterns & Design Principles
- Layered pipeline: input → transformation → model → output
- Plugin/registry pattern for:
- Architectures (dynamic import by name)
- Reader/writer formats in `imageio`
- Trainer variants (via nnUNetTrainer subclasses)
- Clear separation of concerns: conversion, planning, preprocessing, training, inference, evaluation.
- Extensive use of JSON/YAML “plans” for configuration, enabling reproducibility.

##### Technologies & Dependencies
- Python 3.x, PyTorch
- nnU-Net v2 (as a library dependency)
- nibabel, SimpleITK, tifffile for I/O
- HPC scheduler interface (LSF scripts in `batch_running`)
- Standard Python CLI via setuptools entry points

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestration Layer
    subgraph "Orchestration" 
        direction TB
        CLI["CLI / Entry Points"]:::orchestration
        Batch["Batch Running"]:::orchestration
    end

    %% Data Preparation Layer
    subgraph "Data Preparation" 
        direction TB
        Conv["Dataset Conversion"]:::transform
        Plan["Experiment Planning"]:::transform
        Preproc["Preprocessing"]:::transform
    end

    %% Training Layer
    subgraph "Training & Modeling"
        direction TB
        Arch["Model Architectures"]:::compute
        Train["Training"]:::compute
    end

    %% Inference & Evaluation Layer
    subgraph "Inference & Evaluation"
        direction TB
        Inference["Inference"]:::compute
        Post["Postprocessing"]:::transform
        Eval["Evaluation"]:::compute
        Ensemble["Ensembling"]:::compute
        Share["Model Sharing"]:::compute
    end

    %% Utilities Layer
    subgraph "Utilities & I/O"
        direction TB
        Utils["Utilities & I/O Registry"]:::utility
    end

    %% External Dependencies
    NNUNET(["nnU-Net v2"]):::external
    PYT(["PyTorch"]):::external
    HPC(["HPC Scheduler"]):::external
    IOLIB(["I/O Libraries"]):::external

    %% Connections
    CLI -->|runs| Conv
    CLI -->|runs| Plan
    CLI -->|runs| Train
    CLI -->|runs| Inference
    CLI -->|runs| Share

    Batch -->|wraps| CLI
    Batch -->|submits jobs to| HPC

    Conv -->|outputs dataset.json| Plan
    Plan -->|outputs plan.json| Preproc
    Preproc -->|produces preprocessed data| Train
    Train -->|outputs model folds| Inference
    Inference -->|outputs raw predictions| Post
    Post -->|cleans predictions| Eval
    Eval -->|metrics| Ensemble
    Ensemble -->|final segmentations| Share

    Arch -->|provides networks to| Train
    NNUNET -->|dependency| Train
    PYT -->|dependency| Train
    IOLIB -->|used by| Conv
    IOLIB -->|used by| Preproc
    IOLIB -->|used by| Inference

    %% Click Events for component mapping
    click CLI "https://github.com/mic-dkfz/multitalent/blob/public_MT/setup.py"
    click CLI "https://github.com/mic-dkfz/multitalent/blob/public_MT/pyproject.toml"
    click CLI "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/run/run_training.py"
    click CLI "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/run/run_training_MT.py"
    click CLI "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/run/load_pretrained_weights.py"
    click CLI "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/run/load_pretrained_weights_MT.py"
    click CLI "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/model_sharing/entry_points.py"

    click Conv "https://github.com/mic-dkfz/multitalent/tree/public_MT/multitalent/dataset_conversion/"
    click Conv "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/dataset_conversion/generate_dataset_json.py"

    click Plan "https://github.com/mic-dkfz/multitalent/tree/public_MT/multitalent/experiment_planning/"
    click Plan "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/experiment_planning/plan_and_preprocess_entrypoints.py"
    click Plan "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/experiment_planning/plan_and_preprocess_api.py"

    click Preproc "https://github.com/mic-dkfz/multitalent/tree/public_MT/multitalent/preprocessing/"
    click Preproc "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/preprocessing/cropping/cropping.py"
    click Preproc "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/preprocessing/normalization/default_normalization_schemes.py"
    click Preproc "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/preprocessing/preprocessors/default_preprocessor.py"
    click Preproc "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/preprocessing/resampling/resample_torch.py"

    click Arch "https://github.com/mic-dkfz/multitalent/tree/public_MT/multitalent/architectures/"
    click Arch "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/architectures/MedNeXt/MedNeXt.py"
    click Arch "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/architectures/MedNeXt/blocks.py"
    click Arch "https://github.com/mic-dkfz/multitalent/tree/public_MT/multitalent/network_architecture/"
    click Arch "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/network_architecture/hrnet/hrnet.py"

    click Train "https://github.com/mic-dkfz/multitalent/tree/public_MT/multitalent/training/"
    click Train "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/training/lr_scheduler/polylr.py"
    click Train "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/training/nnUNetTrainer/nnUNetTrainer.py"
    click Train "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/training/nnUNetTrainer/project_specific/MultiTalent_native/MultiTalent_trainer.py"

    click Inference "https://github.com/mic-dkfz/multitalent/tree/public_MT/multitalent/inference/"
    click Inference "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/inference/predict_from_raw_data.py"
    click Inference "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/inference/sliding_window_prediction.py"
    click Inference "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/inference/export_prediction.py"
    click Inference "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/inference/data_iterators.py"

    click Post "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/postprocessing/remove_connected_components.py"

    click Eval "https://github.com/mic-dkfz/multitalent/tree/public_MT/multitalent/evaluation/"
    click Eval "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/evaluation/evaluate_predictions.py"
    click Eval "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/evaluation/accumulate_cv_results.py"
    click Eval "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/ensembling/ensemble.py"

    click Batch "https://github.com/mic-dkfz/multitalent/tree/public_MT/multitalent/batch_running/"
    click Batch "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/batch_running/benchmarking/generate_benchmarking_commands.py"
    click Batch "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/batch_running/benchmarking/summarize_benchmark_results.py"
    click Batch "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/batch_running/generate_lsf_runs_customDecathlon.py"
    click Batch "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/batch_running/collect_results_custom_Decathlon.py"

    click Share "https://github.com/mic-dkfz/multitalent/tree/public_MT/multitalent/model_sharing/"
    click Share "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/model_sharing/model_download.py"
    click Share "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/model_sharing/model_export.py"
    click Share "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/model_sharing/model_import.py"

    click Utils "https://github.com/mic-dkfz/multitalent/tree/public_MT/multitalent/utilities/"
    click Utils "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/utilities/file_path_utilities.py"
    click Utils "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/utilities/find_class_by_name.py"
    click Utils "https://github.com/mic-dkfz/multitalent/tree/public_MT/multitalent/imageio/"
    click Utils "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/imageio/reader_writer_registry.py"
    click Utils "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/imageio/nibabel_reader_writer.py"
    click Utils "https://github.com/mic-dkfz/multitalent/blob/public_MT/multitalent/imageio/simpleitk_reader_writer.py"

    %% Legend
    subgraph Legend
        direction LR
        T["Transform (blue)"]:::transform
        C["Compute (green)"]:::compute
        O["Orchestration (orange)"]:::orchestration
        U["Utility (gray)"]:::utility
        E["External"]:::external
    end

    %% Styles
    classDef transform fill:#cce5ff,stroke:#004085,color:#004085;
    classDef compute fill:#d4edda,stroke:#155724,color:#155724;
    classDef orchestration fill:#fff3cd,stroke:#856404,color:#856404;
    classDef utility fill:#e2e3e5,stroke:#6c757d,color:#6c757d;
    classDef external stroke-dasharray: 5 5,stroke:#343a40,color:#343a40;
```
