# microsoft/biomedparse architecture

> Architecture diagram of the GitHub repository microsoft/biomedparse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/biomedparse
- Repository: https://github.com/microsoft/biomedparse

## Overview

##### Project Type and Purpose
- That this is a research/deployment system for biomedical image analysis based on a unified foundation model.
- From the README that it supports segmentation, detection, and recognition and is constructed to handle a wide range of imaging modalities used in biomedicine.
- That the project integrates data preprocessing, training/inference pipelines, and modular model components backed by extensive configuration files and utility scripts.

##### Major System Components
a. Data Management and Preprocessing:
– Include modules for dataset creation, mapping, registration, and evaluation.
– Represent directories like “biomedparse_datasets”, “datasets” (with submodules for dataset mappers, evaluation, registration, visual sampling), and scripts for creating customer datasets.
– Show the data sources (raw images, masks, JSON files) and how these are converted into training and testing sets (the “create-customer-datasets.py” script and registration files).

b. Model Architecture and Building:
– Display the “modeling” component as the core model-building block.
– Within the “modeling” folder, highlight submodules such as “BaseModel”, “architectures” (which include several versions of the main model, attention mechanisms, and backbone modules), “language” (with encoder and transformer support), “modules” (attention, criterion, matcher, and postprocessing), and “vision” (with backbone and encoder support).
– Indicate that configuration files from the “configs” folder (like biomed_seg_lang_v1.yaml and biomedparse_inference.yaml) drive model settings.

c. Training and Distributed Training:
– Show the “trainer” directory that holds various training scripts, including a default trainer, distributed trainer, and specific trainers for the XDecoder model.
– Connect this component to the configuration from “configs” and also link it to utilities and hooks in “trainer/utils”. This part manages model checkpointing and logs training progress.

d. Inference and Pipeline:
– Represent the “pipeline” and “inference_utils” directories. The pipeline module (e.g., XDecoderPipeline.py) orchestrates data flow during inference.
– Indicate how “inference_utils” provides the inference routines, output processing, and additional support code for evaluating segmentation, detection, and recognition.
– Include connections to example notebooks and scripts like “example_prediction.py” that demonstrate inference using sample images in the “examples” directory.

e. Utilities and Configuration:
– Include a component for “utilities” that provides support functions for configuration management, argument parsing, distributed processing, and visualization.
– This is a central, cross-cutting concern that links the training, inference, and dataset registration steps.
– Also indicate the “assets” folder which provides external dependency requirements and shell scripts (e.g., train.sh, eval.sh) tying the workflow together.

f. Containerization and Environment Setup:

##### Relationships and Data Flow
- Raw biomedical image data (from datasets and customer data) flows into the data preprocessing and registration stage.
- Processed datasets are consumed by the training component (trainer) where the model defined in the modeling block is trained using configurations.
- Once a model checkpoint is produced, it is utilized by the inference pipeline that further feeds the sample scripts and notebooks.
- Utility modules and configuration files are used by all major components to maintain consistency.
– Emphasize that the training and inference pipelines are orchestrated via specific shell scripts (train.sh, eval.sh) and notebooks (e.g., inference_examples_*.ipynb).

– Ensure that configuration files and environment set-up scripts are also indicated as they play important roles in system orchestration.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Management & Preprocessing"
        RAW["Raw Biomedical Data"]:::data
        BDS["biomedparse_datasets"]:::data
        DS["datasets (mappers,evaluation,registration,visual_sampler)"]:::data
        CUST["create-customer-datasets.py"]:::script
    end

    %% Modeling Layer
    subgraph "Model Architecture & Building"
        MODEL["modeling (architectures,language,modules,vision)"]:::model
        CONFIG["configs (biomed_seg_lang_v1.yaml,biomedparse_inference.yaml)"]:::utils
    end

    %% Training Layer
    subgraph "Training & Distributed Training"
        TRAINER["trainer (default,distributed,XDecoder)"]:::trainer
        TRAINUTILS["trainer/utils"]:::utils
    end

    %% Inference Layer
    subgraph "Inference & Pipeline"
        PIPELINE["pipeline (XDecoderPipeline.py)"]:::inference
        INFUTILS["inference_utils"]:::inference
        EXAMPLE["example_prediction.py"]:::inference
    end

    %% Utilities and Configuration
    subgraph "Utilities & Scripts"
        UTIL["utilities"]:::utils
        ASSETS["assets/scripts (train.sh,eval.sh)"]:::utils
    end

    %% Containerization
    subgraph "Containerization & Environment"
        DOCKER["docker"]:::container
    end

    %% Data Flow Connections
    RAW --> BDS
    RAW --> DS
    BDS --> CUST
    DS --> CUST

    CUST -- "processed_data" --> TRAINER

    CONFIG -- "configures" --> MODEL
    MODEL -- "supplies_model" --> TRAINER
    CONFIG -- "parameters" --> TRAINER
    UTIL -- "supports" --> TRAINER
    ASSETS -- "orchestrates" --> TRAINER
    TRAINUTILS -- "hooks/checkpoints" --> TRAINER

    TRAINER -- "model_checkpoint" --> PIPELINE
    TRAINER -- "model_checkpoint" --> INFUTILS

    PIPELINE -- "orchestrates" --> INFUTILS
    PIPELINE -- "demonstrates" --> EXAMPLE
    UTIL -- "common_utilities" --> INFUTILS
    UTIL -- "common_utilities" --> TRAINER

    %% Linking Containerization
    UTIL -- "enables_environment" --> DOCKER

    %% Click Events
    click BDS "https://github.com/microsoft/biomedparse/tree/main/biomedparse_datasets/"
    click DS "https://github.com/microsoft/biomedparse/tree/main/datasets/"
    click MODEL "https://github.com/microsoft/biomedparse/tree/main/modeling/"
    click TRAINER "https://github.com/microsoft/biomedparse/tree/main/trainer/"
    click PIPELINE "https://github.com/microsoft/biomedparse/tree/main/pipeline/"
    click INFUTILS "https://github.com/microsoft/biomedparse/tree/main/inference_utils/"
    click EXAMPLE "https://github.com/microsoft/biomedparse/blob/main/example_prediction.py"
    click UTIL "https://github.com/microsoft/biomedparse/tree/main/utilities/"
    click CONFIG "https://github.com/microsoft/biomedparse/tree/main/configs/"
    click ASSETS "https://github.com/microsoft/biomedparse/tree/main/assets/scripts/"
    click DOCKER "https://github.com/microsoft/biomedparse/tree/main/docker/"

    %% Styles
    classDef data fill:#ffcccc,stroke:#cc0000,stroke-width:2px;
    classDef script fill:#f9e79f,stroke:#d4ac0d,stroke-width:2px;
    classDef model fill:#d5f5e3,stroke:#27ae60,stroke-width:2px;
    classDef trainer fill:#d6eaf8,stroke:#2471a3,stroke-width:2px;
    classDef inference fill:#fdebd0,stroke:#d35400,stroke-width:2px;
    classDef utils fill:#ebedef,stroke:#7f8c8d,stroke-width:2px;
    classDef container fill:#fcf3cf,stroke:#b7950b,stroke-width:2px;
```
