How project-monai/vlm-radiology-agent-framework works
Project Type and Purpose
• This is a full-stack machine-learning pipeline and demo application for VILA-M3, a vision-language model tailored to medical imaging.
• It provides:
• Data preparation modules (m3/data_prepare)
• Training scripts and configurations (m3/train)
• Evaluation harness (m3/eval)
• A containerized Gradio demo (m3/demo)
• A REST server for integration into an external DICOM viewer (server-slicer and plugins/RadCoPilot_Slicer)
• Utility notebooks and third-party model code
Key Directories and Architectural Patterns
• m3/data_prepare: modular data pipelines per expert (CXR, BRATS, VQA, etc.)
• m3/train: shell scripts that spawn distributed training jobs of VILA-M3 on GPU clusters
• m3/eval: Slurm job scripts + Python metric scripts for classification, VQA, report generation
• m3/demo: Dockerfile + Gradio app + expert wrapper classes (plugin pattern for expert models)
• server-slicer + plugins/RadCoPilot_Slicer: a microservice (Python HTTP server) + a plugin for medical image software
• notebooks: one-off analysis and demo notebooks
• thirdparty/VILA: external VILA code integrated as a subtree module
Main Components
a. Data Preparation Service
• Input: raw medical image datasets (MIMIC-CXR, BRATS, PathVQA, etc.)
• Modules: data_utils, expert_extract_slices, instruction generators
• Output: formatted JSON/Parquet data for training and evaluation
b. Training Orchestrator
• Shell scripts (m3-vila3b.sh, m3-vila8b.sh, m3-vila13b.sh)
• Launches distributed PyTorch training jobs on GPUs
• Consumes prepared data, writes model checkpoints to storage (HuggingFace or local)
c. Evaluation Harness
• Slurm scripts trigger inference via Python scripts under m3/eval/scripts
• Metrics computed per task (classification, VQA, report generation)
• Results aggregated and stored for analysis
d. Demo & Deployment
• Docker container built from m3/demo/Dockerfile
• Gradio front-end (gradio_m3.py)
• Expert adapter classes instantiate segmentation/classification experts
• Fetches VILA-M3 checkpoint from HuggingFace or local path
• Exposes UI at localhost:7860
e. Integration Server (server-slicer)
• vila-m3-server.py: a Flask/FastAPI-style HTTP server
• utils.py: image preprocessing, label mapping
• label_groups_dict.json: configured expert label groups
• Exposes REST endpoints for inference on DICOM slices/volumes
f. Client Plugin (RadCoPilot_Slicer)
• C++/Qt plugin with UI definitions (.ui files)
• Connects to the server-slicer via HTTP (client.py)
• Embeds inference results back into the medical imaging application
Relationships and Data Flow
• Preprocessed Data → Training Orchestrator → Model Checkpoints → Evaluation Harness
• Model Checkpoints + Expert Models → Demo Container → Gradio UI
• Model Checkpoints → server-slicer → REST API
• Plugin (RadCoPilot_Slicer) ↔ REST API → DICOM application
Architectural Patterns and Design Principles
• Modular pipelines: each expert or task has its own subdirectory and interface
• Separation of concerns: data prep, training, evaluation, demo, integration are distinct layers
• Microservice architecture for real-time inference (server-slicer)
• Plugin architecture for third-party medical imaging software
• Containerization for reproducibility (Docker)