How duyhominhnguyen/lvm-med works
Project Type and Purpose
• This is an open-source research toolkit (“LVM-Med”) for training, fine-tuning, and evaluating large-scale self-supervised vision models on medical imaging tasks.
• It provides downstream pipelines for 2D/3D segmentation, image classification, object detection, prompt-based segmentation (via SAM/MedSAM), and ONNX export.
High-Level Architectural Components
a. Entry-point scripts (CLI)
• train_segmentation.py, train_classification.py, medsam.py, zero_shot_segmentation.py, evaluate.py, prepare_dataset.py, convert_to_coco.py
• Notebooks (Prompt_Demo.ipynb) for interactive demos
b. Configuration & Metadata
• YAML config files in dataloader/yaml_data
• JSON/PKL splits in files_split, dataset split scripts
c. Data Pipeline
• prepare_dataset.py & dataset_demo for gathering raw files
• dataloader/ (dataloader.py, dataset.py, transforms, sam_transforms)
• datasets_split & files_split for train/test splits
d. Model Zoo & Definitions
• model/ (unet_vit.py, unet_vit_classification.py)
• segment_anything/ integration (builder, sam.py, predictor, transformers, onnx helpers)
• classification_R50 & classification_VIT training routines
• segmentation_2d & segmentation_3d task scripts
• object_detection/ (mmdetection-style configs, training & conversion scripts)
• further_training_lvm_med for additional pretraining on large datasets
e. Training & Evaluation Engine
• Torch-based training loops reading config → data loader → model → optimizer/scheduler → checkpoint/log
• Evaluation via evaluate.py or test flags in CLI scripts
f. Model Serialization & Export
• PyTorch checkpoints in working_dir/checkpoint/
• ONNX export routines in onnx_model/
g. External Dependencies
• PyTorch (+ CUDA), torchvision
• segmentation_models.pytorch (U-Net, DeepLab, etc.)
• Facebook’s Segment Anything (SAM) & MedSAM
• MMDetection for object detection backends
• ONNX for model interoperability
Relationships & Data Flow
• Entry-point script
↓ reads
• YAML/CLI config
↓ instantiates
• Dataset (dataloader + transforms)
↓ yields
• Batches of images & annotations
↓ fed into
• Model (backbone + task head)
↓ optimized by
• Trainer (loss computation, backprop, scheduler)
↓ produces
• Checkpoints & logs
↓ optionally exported to
• ONNX format
Architectural Patterns & Principles
• Modular pipelines per downstream task (plug-and-play backbones & heads)
• Config-driven orchestration (YAML + CLI flags)
• Separation of concerns:
• Data loading & preprocessing
• Model definition
• Training/evaluation logic
• Serialization/export
• Extensibility via external libraries (segmentation_models.pytorch, MMDetection, SAM)
Detailing Essentials
• Relationships between config files, data loaders, model definitions, and training scripts.
• Role of external frameworks and where they integrate (e.g., MMDetection under object_detection, SAM under segment_anything).
• Where weights live (lvm_med_weights, working_dir/checkpoint) and where logs are stored.
• Interactions with external dataset repositories (e.g., Kaggle, VinDr) via prepare_dataset and convert_to_coco scripts.