How mirthai/med3dvlm works
Project type and purpose
– Med3DVLM is a research-oriented Vision-Language Model for 3D medical image analysis.
– It consists of data preparation, model training (contrastive and multi-modal), evaluation, and demo/inference (CLI & a small web app).
Break down the main components
A. Data Layer
• Raw datasets (M3D-Cap, M3D-VQA) in data/ (NIfTI files + CSVs)
• Data-prep scripts in src/utils (e.g., m3d_cap_data_prepare_128.py, rename_csv.py)
B. Dataset APIs
• src/dataset/clip_dataset.py and mllm_dataset.py load NIfTI volumes + text
• src/dataset/prompt_templates.py holds textual prompts
C. Model Layer
• Encoder: src/model/encoder/ (DCFormer, ViT builder)
• Projector: src/model/projector/ (MLP, MHSA)
• LLM adapter: src/model/llm/qwen.py
• Overall VLM architecture: src/model/vlm_arch.py
• Contrastive CLIP head: src/model/CLIP.py
D. Training & Pre-training
• Scripts in src/train/ (train_clip.py, train_vlm.py, trainer.py)
• Shell wrappers in scripts/ (train_clip.sh, pretrain_mm_projector.sh, finetune_lora.sh, merge_lora_weights_and_save_hf_model.sh)
E. Evaluation
• Python entry points in src/eval/ (eval_caption.py, eval_clip.py, eval_vqa.py)
• Shell wrappers in scripts/eval/
F. Demo & Serving
• CLI demo: src/demo/demo.py
• Minimal web app: app.py (likely Flask/FastAPI)
G. Configuration & Environment
• env.yaml, requirements.txt, .gitignore
Relationships and data flow
– Raw data → data-prep utils → preprocessed NIfTI + CSV → dataset loaders → model backbones/projectors → loss heads → output model weights
– Model weights → trainer.py → saved checkpoints in output/ or models/
– Checkpoints → evaluation scripts → metrics (R@1, METEOR, VQA acc)
– Final model + weights → CLI demo or app.py web server → user queries
Architectural patterns
– Modular ML pipeline: clear separation of data, model, training, evaluation, and serving layers
– Two-stage training: 1) contrastive (SigLIP) 2) multi-modal pre-training and LoRA fine-tuning
– Dual-stream MLP-Mixer projector fusing low/high-level features + text embeddings
– Decomposed 3D convs in DCFormer for efficient volumetric encoding
Technologies & frameworks
– PyTorch 2.6.0 + TorchVision + MONAI for 3D medical images
– Deepspeed for large-scale training
– Conda/Pip for environment (env.yaml, requirements.txt)
– NIfTI file format, HuggingFace datasets, ModelScope
Detailed element checklist
– Component name labels (e.g., “DCFormer Encoder”)
– Notes on patterns (e.g., “Decomposed 3D convs → reduced memory”)