ai-14/micar-vl-moe

How ai-14/micar-vl-moe works

MicarVLMoE is a medical image captioning and report-generation research system. Its main workflow starts with dataset-specific preprocessing and configured train/validation/test data, then tokenizes reports, trains the vision-language model, generates captions, and evaluates them. The model combines multiscale visual encoding, gated cross-modal alignment, and an MoE decoder. The README documents four supported datasets and inference from checkpoints; sampled code shows the training and evaluation path. Unsampled preprocessing and notebook details are represented only at the documented subsystem level.

Main entry points: caption_execute.py Train and test · main.py Runtime entry · models.py Caption model · services.py Experiment services

Components 22

Data preparation

COVCTR prep preprocessors/covctr.py
MMRetinal prep preprocessors/mmretinal.py
PGROSS prep preprocessors/pgross.py
ROCO prep preprocessors/roco.py
Report tokenizer utils/tokenizers.py
Caption dataset utils/datasets.py
Data loaders utils/dataloaders.py

Experiment runtime

Experiment config configs/cfg.py
Runtime entry main.py
Experiment services utils/services.py
Train and test train_test/caption_execute.py
Checkpoint inference inference.ipynb

Caption model

Caption model architecture/models.py
Multiscale vision architecture/blocks.py
Gated alignment architecture/blocks.py
MoE decoder architecture/blocks.py
Causal masking architecture/utils.py

Evaluation and outputs

Training loss utils/losses.py
Caption metrics utils/metrics.py
Reports and checkpoints configs/cfg.py

Other components

Experimenter
Medical datasets

Connections 27

Experimenter → Runtime entry: starts run main.py
Experimenter → COVCTR prep: runs preprocessing README.md
Experimenter → MMRetinal prep: runs preprocessing README.md
Experimenter → PGROSS prep: runs preprocessing README.md
Experimenter → ROCO prep: runs preprocessing README.md
Medical datasets → COVCTR prep: provides data README.md
Medical datasets → MMRetinal prep: provides data README.md
Medical datasets → PGROSS prep: provides data README.md
Medical datasets → ROCO prep: provides data README.md
Runtime entry → Experiment config: parses settings main.py
Runtime entry → Experiment services: initializes run main.py
Runtime entry → Report tokenizer: creates tokenizer main.py
Runtime entry → Train and test: executes training main.py
Train and test → Data loaders: builds splits train_test/caption_execute.py
Data loaders → Caption dataset: constructs datasets utils/dataloaders.py
Caption dataset → Report tokenizer: encodes reports utils/datasets.py
Train and test → Caption model: trains and generates train_test/caption_execute.py
Train and test → Training loss: computes loss train_test/caption_execute.py
Train and test → Caption metrics: scores captions train_test/caption_execute.py
Train and test → Experiment services: saves snapshots train_test/caption_execute.py
Caption model → Multiscale vision: encodes images architecture/models.py
Caption model → Gated alignment: aligns modalities README.md
Caption model → MoE decoder: decodes captions README.md
Caption model → Causal masking: creates masks architecture/models.py
Train and test → Reports and checkpoints: writes results train_test/caption_execute.py
Experimenter → Checkpoint inference: uses notebook README.md
Checkpoint inference → Experiment services: loads checkpoint

Drawn by GitDiagram on November 9, 2026 from the repository's file tree, README and source files. Read as Markdown ·