# duyhominhnguyen/lvm-med architecture

> Architecture diagram of the GitHub repository duyhominhnguyen/lvm-med, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/duyhominhnguyen/lvm-med
- Repository: https://github.com/duyhominhnguyen/lvm-med

## Overview

##### Project Type and Purpose
- This is an open-source research toolkit (“LVM-Med”) for training, fine-tuning, and evaluating large-scale self-supervised vision models on medical imaging tasks.
- It provides downstream pipelines for 2D/3D segmentation, image classification, object detection, prompt-based segmentation (via SAM/MedSAM), and ONNX export.

##### High-Level Architectural Components
a. Entry-point scripts (CLI)
- train_segmentation.py, train_classification.py, medsam.py, zero_shot_segmentation.py, evaluate.py, prepare_dataset.py, convert_to_coco.py
- Notebooks (Prompt_Demo.ipynb) for interactive demos
b. Configuration & Metadata
- YAML config files in dataloader/yaml_data
- JSON/PKL splits in files_split, dataset split scripts
c. Data Pipeline
- prepare_dataset.py & dataset_demo for gathering raw files
- dataloader/ (dataloader.py, dataset.py, transforms, sam_transforms)
- datasets_split & files_split for train/test splits
d. Model Zoo & Definitions
- model/ (unet_vit.py, unet_vit_classification.py)
- segment_anything/ integration (builder, sam.py, predictor, transformers, onnx helpers)
- classification_R50 & classification_VIT training routines
- segmentation_2d & segmentation_3d task scripts
- object_detection/ (mmdetection-style configs, training & conversion scripts)
- further_training_lvm_med for additional pretraining on large datasets
e. Training & Evaluation Engine
- Torch-based training loops reading config → data loader → model → optimizer/scheduler → checkpoint/log
- Evaluation via evaluate.py or test flags in CLI scripts
f. Model Serialization & Export
- PyTorch checkpoints in working_dir/checkpoint/
- ONNX export routines in onnx_model/
g. External Dependencies
- PyTorch (+ CUDA), torchvision
- segmentation_models.pytorch (U-Net, DeepLab, etc.)
- Facebook’s Segment Anything (SAM) & MedSAM
- MMDetection for object detection backends
- ONNX for model interoperability

##### Relationships & Data Flow
- Entry-point script
↓ reads
- YAML/CLI config
↓ instantiates
- Dataset (dataloader + transforms)
↓ yields
- Batches of images & annotations
↓ fed into
- Model (backbone + task head)
↓ optimized by
- Trainer (loss computation, backprop, scheduler)
↓ produces
- Checkpoints & logs
↓ optionally exported to
- ONNX format

##### Architectural Patterns & Principles
- Modular pipelines per downstream task (plug-and-play backbones & heads)
- Config-driven orchestration (YAML + CLI flags)
- Separation of concerns:
- Data loading & preprocessing
- Model definition
- Training/evaluation logic
- Serialization/export
- Extensibility via external libraries (segmentation_models.pytorch, MMDetection, SAM)

##### Detailing Essentials
- Relationships between config files, data loaders, model definitions, and training scripts.
- Role of external frameworks and where they integrate (e.g., MMDetection under object_detection, SAM under segment_anything).
- Where weights live (lvm_med_weights, working_dir/checkpoint) and where logs are stored.
- Interactions with external dataset repositories (e.g., Kaggle, VinDr) via prepare_dataset and convert_to_coco scripts.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Configuration Layer
    subgraph "Configuration Layer"
        direction TB
        YAML["YAML Config"]:::core
        JSON_SPLITS["JSON/PKL Splits"]:::core
        DATASET_SPLITS["Dataset Split Scripts"]:::core
    end

    %% Data Layer
    subgraph "Data Layer"
        direction TB
        PREPARE_DEMO["Dataset Demo Module"]:::core
        PREPARE_SCRIPT["prepare_dataset.py"]:::core
        DATALOADER["PyTorch Dataloader"]:::core
        DATASET_RAW["Raw Dataset Files"]:::datasource
    end

    %% Model Layer
    subgraph "Model Layer"
        direction TB
        UNET_VIT["U-Net + ViT Models"]:::core
        SAM_INTEGRATION["SAM Integration"]:::core
        CLASS_R50["ResNet-50 Classification"]:::core
        CLASS_VIT["ViT Classification"]:::core
        SEG2D["2D Segmentation Pipelines"]:::core
        SEG3D["3D Segmentation Pipelines"]:::core
        OBJ_DET["Object Detection (MMDetection)"]:::core
        FURTHER_PRETRAIN["Self-supervised Pretraining"]:::core
    end

    %% Training / Evaluation Layer
    subgraph "Training/Evaluation Layer"
        direction TB
        SCRIPTS["Entry-point CLI Scripts"]:::core
        EVALUATE["evaluate.py"]:::core
        NOTEBOOK["Prompt Demo Notebook"]:::optional
    end

    %% Storage / Export Layer
    subgraph "Storage/Export Layer"
        direction TB
        CHECKPOINTS["PyTorch Checkpoints"]:::external
        ONNX_EXPORT["ONNX Export Routines"]:::core
        ONNX_MODELS["Prebuilt ONNX Models"]:::external
    end

    %% External Dependencies
    subgraph "External Libraries"
        direction TB
        PYTORCH["PyTorch & CUDA"]:::external
        TORCHVISION["Torchvision"]:::external
        SMLP["segmentation_models.pytorch"]:::external
        SAM_LIB["Facebook SAM & MedSAM"]:::external
        MMDET["MMDetection"]:::external
        ONNX_LIB["ONNX"]:::external
    end

    %% Dataset Sources
    subgraph "Dataset Sources"
        direction TB
        LVM_DATA["lvm-med-training-data"]:::datasource
        KAGGLE["Kaggle / VinDr / Brain / FGADR"]:::datasource
    end

    %% Relationships
    SCRIPTS -->|"reads"| YAML
    SCRIPTS -->|"reads"| JSON_SPLITS
    SCRIPTS -->|"reads"| DATASET_SPLITS

    YAML -->|"instantiates"| DATALOADER
    JSON_SPLITS --> DATALOADER
    DATASET_SPLITS --> PREPARE_SCRIPT

    PREPARE_SCRIPT -->|"prepares"| DATASET_RAW
    PREPARE_DEMO -->|"demos"| DATASET_RAW
    DATASET_RAW -->|"yields batches"| DATALOADER

    DATALOADER -->|"feeds"| UNET_VIT
    DATALOADER -->|"feeds"| CLASS_R50
    DATALOADER -->|"feeds"| CLASS_VIT
    DATALOADER -->|"feeds"| SEG2D
    DATALOADER -->|"feeds"| SEG3D
    DATALOADER -->|"feeds"| OBJ_DET

    UNET_VIT -->|"trained by"| SCRIPTS
    CLASS_R50 -->|"trained by"| SCRIPTS
    CLASS_VIT -->|"trained by"| SCRIPTS
    SEG2D -->|"trained by"| SCRIPTS
    SEG3D -->|"trained by"| SCRIPTS
    OBJ_DET -->|"trained by"| SCRIPTS
    FURTHER_PRETRAIN -->|"uses"| UNET_VIT

    SCRIPTS -->|"calls"| EVALUATE

    SCRIPTS -->|"exports"| CHECKPOINTS
    SCRIPTS -->|"exports"| ONNX_EXPORT
    ONNX_EXPORT -->|"generates"| ONNX_MODELS

    SCRIPTS -->|"imports"| PYTORCH
    SCRIPTS -->|"imports"| TORCHVISION
    SCRIPTS -->|"imports"| SMLP
    SCRIPTS -->|"imports"| SAM_LIB
    SCRIPTS -->|"imports"| MMDET
    SCRIPTS -->|"imports"| ONNX_LIB

    PREPARE_SCRIPT -->|"accesses"| LVM_DATA
    PREPARE_SCRIPT -->|"accesses"| KAGGLE

    %% Click Events
    click YAML "https://github.com/duyhominhnguyen/lvm-med/tree/main/dataloader/yaml_data/"
    click JSON_SPLITS "https://github.com/duyhominhnguyen/lvm-med/tree/main/files_split/"
    click DATASET_SPLITS "https://github.com/duyhominhnguyen/lvm-med/tree/main/datasets_split/"
    click PREPARE_SCRIPT "https://github.com/duyhominhnguyen/lvm-med/blob/main/prepare_dataset.py"
    click PREPARE_DEMO "https://github.com/duyhominhnguyen/lvm-med/tree/main/dataset_demo/"
    click DATALOADER "https://github.com/duyhominhnguyen/lvm-med/blob/main/dataloader/dataloader.py"
    click DATALOADER "https://github.com/duyhominhnguyen/lvm-med/blob/main/dataloader/dataset.py"
    click DATALOADER "https://github.com/duyhominhnguyen/lvm-med/blob/main/dataloader/dataset_ete.py"
    click DATALOADER "https://github.com/duyhominhnguyen/lvm-med/blob/main/dataloader/sam_transforms.py"
    click UNET_VIT "https://github.com/duyhominhnguyen/lvm-med/blob/main/model/unet_vit.py"
    click UNET_VIT "https://github.com/duyhominhnguyen/lvm-med/blob/main/model/unet_vit_classification.py"
    click SAM_INTEGRATION "https://github.com/duyhominhnguyen/lvm-med/blob/main/segment_anything/build_sam.py"
    click SAM_INTEGRATION "https://github.com/duyhominhnguyen/lvm-med/blob/main/segment_anything/modeling/sam.py"
    click SAM_INTEGRATION "https://github.com/duyhominhnguyen/lvm-med/blob/main/segment_anything/modeling/prompt_encoder.py"
    click SAM_INTEGRATION "https://github.com/duyhominhnguyen/lvm-med/blob/main/segment_anything/modeling/mask_decoder.py"
    click SAM_INTEGRATION "https://github.com/duyhominhnguyen/lvm-med/blob/main/segment_anything/predictor.py"
    click SAM_INTEGRATION "https://github.com/duyhominhnguyen/lvm-med/blob/main/segment_anything/utils/onnx.py"
    click CLASS_R50 "https://github.com/duyhominhnguyen/lvm-med/blob/main/classification_R50/train_R50_classification.py"
    click CLASS_VIT "https://github.com/duyhominhnguyen/lvm-med/blob/main/classification_VIT/train_SAMVIT_classification.py"
    click SEG2D "https://github.com/duyhominhnguyen/lvm-med/blob/main/segmentation_2d/train_R50_seg_adam_optimizer_2d.py"
    click SEG2D "https://github.com/duyhominhnguyen/lvm-med/blob/main/segmentation_2d/LVMMed_SAM_2d.py"
    click SEG2D "https://github.com/duyhominhnguyen/lvm-med/blob/main/segmentation_2d/MedSAM_2d.py"
    click SEG2D "https://github.com/duyhominhnguyen/lvm-med/blob/main/segmentation_2d/zero_shot_LVMMed_SAM_2d.py"
    click SEG2D "https://github.com/duyhominhnguyen/lvm-med/blob/main/segmentation_2d/zero_shot_SAM_2d.py"
    click SEG3D "https://github.com/duyhominhnguyen/lvm-med/blob/main/segmentation_3d/train_R50_seg_adam_optimizer_3d.py"
    click SEG3D "https://github.com/duyhominhnguyen/lvm-med/blob/main/segmentation_3d/LVMMed_SAM_3d.py"
    click SEG3D "https://github.com/duyhominhnguyen/lvm-med/blob/main/segmentation_3d/MedSAM_3d.py"
    click OBJ_DET "https://github.com/duyhominhnguyen/lvm-med/tree/main/object_detection/"
    click OBJ_DET "https://github.com/duyhominhnguyen/lvm-med/blob/main/object_detection/convert_to_coco.py"
    click FURTHER_PRETRAIN "https://github.com/duyhominhnguyen/lvm-med/tree/main/further_training_lvm_med/"
    click SCRIPTS "https://github.com/duyhominhnguyen/lvm-med/blob/main/train_segmentation.py"
    click SCRIPTS "https://github.com/duyhominhnguyen/lvm-med/blob/main/train_classification.py"
    click SCRIPTS "https://github.com/duyhominhnguyen/lvm-med/blob/main/medsam.py"
    click SCRIPTS "https://github.com/duyhominhnguyen/lvm-med/blob/main/zero_shot_segmentation.py"
    click SCRIPTS "https://github.com/duyhominhnguyen/lvm-med/blob/main/evaluate.py"
    click SCRIPTS "https://github.com/duyhominhnguyen/lvm-med/blob/main/convert_to_coco.py"
    click NOTEBOOK "https://github.com/duyhominhnguyen/lvm-med/blob/main/notebook/Prompt_Demo.ipynb"
    click EVALUATE "https://github.com/duyhominhnguyen/lvm-med/blob/main/evaluate.py"
    click CHECKPOINTS "https://github.com/duyhominhnguyen/lvm-med/tree/main/working_dir/checkpoint/"
    click ONNX_EXPORT "https://github.com/duyhominhnguyen/lvm-med/blob/main/onnx_model/torch2onnx.py"
    click ONNX_EXPORT "https://github.com/duyhominhnguyen/lvm-med/blob/main/onnx_model/load_onnx.py"
    click ONNX_MODELS "https://github.com/duyhominhnguyen/lvm-med/tree/main/onnx_model/"
    click LVM_DATA "https://github.com/duyhominhnguyen/lvm-med/tree/main/lvm-med-training-data/"
    click LVM_DATA "https://github.com/duyhominhnguyen/lvm-med/blob/main/lvm-med-training-data/dataset.md"
    click KAGGLE "https://github.com/duyhominhnguyen/lvm-med/tree/main/lvm-med-training-data/"

    %% Styles
    classDef core fill:#D0E8FF,stroke:#0366d6,stroke-width:1px
    classDef external fill:#DFFFE0,stroke:#2a8f00,stroke-width:1px
    classDef datasource fill:#FFF4D0,stroke:#d48f00,stroke-width:1px
    classDef optional stroke-dasharray: 5 5,fill:#FFEFEF,stroke:#d62728,stroke-width:1px
```
