# project-monai/vlm-radiology-agent-framework architecture

> Architecture diagram of the GitHub repository project-monai/vlm-radiology-agent-framework, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/project-monai/vlm-radiology-agent-framework
- Repository: https://github.com/project-monai/vlm-radiology-agent-framework

## Overview

##### Project Type and Purpose
- This is a full-stack machine-learning pipeline and demo application for VILA-M3, a vision-language model tailored to medical imaging.
- It provides:
- Data preparation modules (m3/data_prepare)
- Training scripts and configurations (m3/train)
- Evaluation harness (m3/eval)
- A containerized Gradio demo (m3/demo)
- A REST server for integration into an external DICOM viewer (server-slicer and plugins/RadCoPilot_Slicer)
- Utility notebooks and third-party model code

##### Key Directories and Architectural Patterns
- m3/data_prepare: modular data pipelines per expert (CXR, BRATS, VQA, etc.)
- m3/train: shell scripts that spawn distributed training jobs of VILA-M3 on GPU clusters
- m3/eval: Slurm job scripts + Python metric scripts for classification, VQA, report generation
- m3/demo: Dockerfile + Gradio app + expert wrapper classes (plugin pattern for expert models)
- server-slicer + plugins/RadCoPilot_Slicer: a microservice (Python HTTP server) + a plugin for medical image software
- notebooks: one-off analysis and demo notebooks
- thirdparty/VILA: external VILA code integrated as a subtree module

##### Main Components
a. Data Preparation Service
- Input: raw medical image datasets (MIMIC-CXR, BRATS, PathVQA, etc.)
- Modules: data_utils, expert_extract_slices, instruction generators
- Output: formatted JSON/Parquet data for training and evaluation

b. Training Orchestrator
- Shell scripts (m3-vila3b.sh, m3-vila8b.sh, m3-vila13b.sh)
- Launches distributed PyTorch training jobs on GPUs
- Consumes prepared data, writes model checkpoints to storage (HuggingFace or local)

c. Evaluation Harness
- Slurm scripts trigger inference via Python scripts under m3/eval/scripts
- Metrics computed per task (classification, VQA, report generation)
- Results aggregated and stored for analysis

d. Demo & Deployment
- Docker container built from m3/demo/Dockerfile
- Gradio front-end (gradio_m3.py)
- Expert adapter classes instantiate segmentation/classification experts
- Fetches VILA-M3 checkpoint from HuggingFace or local path
- Exposes UI at localhost:7860

e. Integration Server (server-slicer)
- vila-m3-server.py: a Flask/FastAPI-style HTTP server
- utils.py: image preprocessing, label mapping
- label_groups_dict.json: configured expert label groups
- Exposes REST endpoints for inference on DICOM slices/volumes

f. Client Plugin (RadCoPilot_Slicer)
- C++/Qt plugin with UI definitions (.ui files)
- Connects to the server-slicer via HTTP (client.py)
- Embeds inference results back into the medical imaging application

##### Relationships and Data Flow
- Preprocessed Data → Training Orchestrator → Model Checkpoints → Evaluation Harness
- Model Checkpoints + Expert Models → Demo Container → Gradio UI
- Model Checkpoints → server-slicer → REST API
- Plugin (RadCoPilot_Slicer) ↔ REST API → DICOM application

##### Architectural Patterns and Design Principles
- Modular pipelines: each expert or task has its own subdirectory and interface
- Separation of concerns: data prep, training, evaluation, demo, integration are distinct layers
- Microservice architecture for real-time inference (server-slicer)
- Plugin architecture for third-party medical imaging software
- Containerization for reproducibility (Docker)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        RawData[(Raw Medical Image Datasets)]:::external
        DataPrep["Data Preparation Service"]:::batch
        DataUtils["data_utils.py"]:::batch
        Experts["Expert Data Pipelines"]:::batch
        FormattedData[(Formatted JSON/Parquet Data)]:::batch
    end

    %% Compute Layer
    subgraph "Compute Layer"
        direction TB
        subgraph "Training Orchestrator"
            direction TB
            TrainShell3["m3-vila3b.sh"]:::batch
            TrainShell8["m3-vila8b.sh"]:::batch
            TrainShell13["m3-vila13b.sh"]:::batch
        end
        Checkpoints[(Model Checkpoints Storage)]:::batch

        subgraph "Evaluation Harness"
            direction TB
            SlurmScripts["Slurm Job Scripts"]:::batch
            EvalScripts["Python Metric Scripts"]:::batch
        end
        Metrics[(Evaluation Metrics Output)]:::batch
    end

    %% Serving Layer
    subgraph "Serving Layer"
        direction TB
        subgraph "Demo & Deployment"
            direction TB
            Docker["Demo Container"]:::realtime
            GradioUI["Gradio Front-end UI"]:::realtime
            Adapters["Demo Expert Adapters"]:::realtime
        end
        subgraph "Integration Server"
            direction TB
            Server["vila-m3-server.py"]:::realtime
            Utils["utils.py"]:::realtime
            Labels["label_groups_dict.json"]:::realtime
        end
    end

    %% Client Layer
    subgraph "Client Layer"
        direction TB
        Plugin["RadCoPilot Slicer Plugin UI"]:::realtime
        ClientMod["client.py"]:::realtime
    end

    %% External Dependencies
    HuggingFace[(HuggingFace Checkpoint Storage)]:::external
    SlurmCluster[(Slurm / GPU Cluster)]:::external

    %% Data Flows
    RawData -->|"raw images"| DataPrep
    DataPrep -->|"formatted data"| FormattedData
    FormattedData -->|"training data"| TrainShell3
    FormattedData -->|"training data"| TrainShell8
    FormattedData -->|"training data"| TrainShell13
    TrainShell3 -->|"checkpoints"| Checkpoints
    TrainShell8 -->|"checkpoints"| Checkpoints
    TrainShell13 -->|"checkpoints"| Checkpoints
    Checkpoints -->|"model ckpt"| SlurmScripts
    SlurmScripts -->|"inference"| EvalScripts
    EvalScripts -->|"metrics"| Metrics

    Checkpoints -->|"checkpoint"| Docker
    Docker -->|"serves UI"| GradioUI
    Adapters -->|"load experts"| Docker

    Checkpoints -->|"checkpoint"| Server
    Server -->|"REST API"| Plugin
    ClientMod -->|"HTTP request/response"| Server
    Plugin -->|"embed results"| Plugin

    %% External interactions
    TrainShell3 -->|"submits to"| SlurmCluster
    SlurmScripts -->|"runs on"| SlurmCluster
    Checkpoints -->|"push/pull"| HuggingFace

    %% Click Events
    click DataPrep "https://github.com/project-monai/vlm-radiology-agent-framework/tree/main/m3/data_prepare/"
    click DataUtils "https://github.com/project-monai/vlm-radiology-agent-framework/blob/main/m3/data_prepare/data_utils.py"
    click Experts "https://github.com/project-monai/vlm-radiology-agent-framework/tree/main/m3/data_prepare/experts/"
    click TrainShell3 "https://github.com/project-monai/vlm-radiology-agent-framework/blob/main/m3/train/m3-vila3b.sh"
    click TrainShell8 "https://github.com/project-monai/vlm-radiology-agent-framework/blob/main/m3/train/m3-vila8b.sh"
    click TrainShell13 "https://github.com/project-monai/vlm-radiology-agent-framework/blob/main/m3/train/m3-vila13b.sh"
    click SlurmScripts "https://github.com/project-monai/vlm-radiology-agent-framework/blob/main/m3/eval/*.slurm"
    click EvalScripts "https://github.com/project-monai/vlm-radiology-agent-framework/tree/main/m3/eval/scripts/"
    click Docker "https://github.com/project-monai/vlm-radiology-agent-framework/tree/main/m3/demo/Dockerfile"
    click GradioUI "https://github.com/project-monai/vlm-radiology-agent-framework/blob/main/m3/demo/gradio_m3.py"
    click Adapters "https://github.com/project-monai/vlm-radiology-agent-framework/tree/main/m3/demo/experts/"
    click Server "https://github.com/project-monai/vlm-radiology-agent-framework/blob/main/server-slicer/vila-m3-server.py"
    click Utils "https://github.com/project-monai/vlm-radiology-agent-framework/blob/main/server-slicer/utils.py"
    click Labels "https://github.com/project-monai/vlm-radiology-agent-framework/blob/main/server-slicer/label_groups_dict.json"
    click Plugin "https://github.com/project-monai/vlm-radiology-agent-framework/tree/main/plugins/RadCoPilot_Slicer/"
    click ClientMod "https://github.com/project-monai/vlm-radiology-agent-framework/blob/main/plugins/RadCoPilot_Slicer/RadCoPilotLib/client.py"
    click HuggingFace "https://github.com/project-monai/vlm-radiology-agent-framework/tree/main/thirdparty/VILA/"

    %% Styles
    classDef batch fill:#E0F7FA,stroke:#006064,color:#006064
    classDef realtime fill:#E8F5E9,stroke:#2E7D32,color:#2E7D32
    classDef external fill:#FFF3E0,stroke:#EF6C00,color:#E65100
```
