# ai-14/micar-vl-moe architecture

> Architecture diagram of the GitHub repository ai-14/micar-vl-moe, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ai-14/micar-vl-moe
- Repository: https://github.com/ai-14/micar-vl-moe

## Overview

MicarVLMoE is a medical image captioning and report-generation research system. Its main workflow starts with dataset-specific preprocessing and configured train/validation/test data, then tokenizes reports, trains the vision-language model, generates captions, and evaluates them. The model combines multiscale visual encoding, gated cross-modal alignment, and an MoE decoder. The README documents four supported datasets and inference from checkpoints; sampled code shows the training and evaluation path. Unsampled preprocessing and notebook details are represented only at the documented subsystem level.

Main entry points: [`train_test/caption_execute.py`](https://github.com/ai-14/micar-vl-moe/blob/master/train_test/caption_execute.py) (Train and test), [`main.py`](https://github.com/ai-14/micar-vl-moe/blob/master/main.py) (Runtime entry), [`architecture/models.py`](https://github.com/ai-14/micar-vl-moe/blob/master/architecture/models.py) (Caption model), [`utils/services.py`](https://github.com/ai-14/micar-vl-moe/blob/master/utils/services.py) (Experiment services)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_covprep["COVCTR prep<br/>[covctr.py]"]
  node_retprep["MMRetinal prep<br/>[mmretinal.py]"]
  node_grossprep["PGROSS prep<br/>[pgross.py]"]
  node_rocoprep["ROCO prep<br/>[roco.py]"]
  node_tokenizer["Report tokenizer<br/>[tokenizers.py]"]
  node_dataset["Caption dataset<br/>[datasets.py]"]
  node_loaders["Data loaders<br/>[dataloaders.py]"]
end

subgraph group_runtime["Experiment runtime"]
  node_config["Experiment config<br/>[cfg.py]"]
  node_main["Runtime entry<br/>[main.py]"]
  node_services["Experiment services<br/>[services.py]"]
  node_trainer["Train and test<br/>[caption_execute.py]"]
  node_inference["Checkpoint inference<br/>[inference.ipynb]"]
end

subgraph group_model["Caption model"]
  node_captionmodel["Caption model<br/>[models.py]"]
  node_vision["Multiscale vision<br/>[blocks.py]"]
  node_alignment["Gated alignment<br/>[blocks.py]"]
  node_moe["MoE decoder<br/>[blocks.py]"]
  node_causalmask["Causal masking<br/>[utils.py]"]
end

subgraph group_evaluation["Evaluation and outputs"]
  node_loss["Training loss<br/>[losses.py]"]
  node_metrics["Caption metrics<br/>[metrics.py]"]
  node_artifacts["Reports and checkpoints<br/>[cfg.py]"]
end

node_researcher(("Experimenter"))
node_datasets[("Medical datasets")]

node_researcher -->|"starts run"| node_main
node_researcher -->|"runs preprocessing"| node_covprep
node_researcher -->|"runs preprocessing"| node_retprep
node_researcher -->|"runs preprocessing"| node_grossprep
node_researcher -->|"runs preprocessing"| node_rocoprep
node_datasets -->|"provides data"| node_covprep
node_datasets -->|"provides data"| node_retprep
node_datasets -->|"provides data"| node_grossprep
node_datasets -->|"provides data"| node_rocoprep
node_main -->|"parses settings"| node_config
node_main -->|"initializes run"| node_services
node_main -->|"creates tokenizer"| node_tokenizer
node_main -->|"executes training"| node_trainer
node_trainer -->|"builds splits"| node_loaders
node_loaders -->|"constructs datasets"| node_dataset
node_dataset -->|"encodes reports"| node_tokenizer
node_trainer -->|"trains and generates"| node_captionmodel
node_trainer -->|"computes loss"| node_loss
node_trainer -->|"scores captions"| node_metrics
node_trainer -->|"saves snapshots"| node_services
node_captionmodel -->|"encodes images"| node_vision
node_captionmodel -->|"aligns modalities"| node_alignment
node_captionmodel -->|"decodes captions"| node_moe
node_captionmodel -->|"creates masks"| node_causalmask
node_trainer -->|"writes results"| node_artifacts
node_researcher -->|"uses notebook"| node_inference
node_inference -.->|"loads checkpoint"| node_services

click node_covprep "https://github.com/ai-14/micar-vl-moe/blob/master/preprocessors/covctr.py"
click node_retprep "https://github.com/ai-14/micar-vl-moe/blob/master/preprocessors/mmretinal.py"
click node_grossprep "https://github.com/ai-14/micar-vl-moe/blob/master/preprocessors/pgross.py"
click node_rocoprep "https://github.com/ai-14/micar-vl-moe/blob/master/preprocessors/roco.py"
click node_config "https://github.com/ai-14/micar-vl-moe/blob/master/configs/cfg.py"
click node_main "https://github.com/ai-14/micar-vl-moe/blob/master/main.py"
click node_services "https://github.com/ai-14/micar-vl-moe/blob/master/utils/services.py"
click node_tokenizer "https://github.com/ai-14/micar-vl-moe/blob/master/utils/tokenizers.py"
click node_dataset "https://github.com/ai-14/micar-vl-moe/blob/master/utils/datasets.py"
click node_loaders "https://github.com/ai-14/micar-vl-moe/blob/master/utils/dataloaders.py"
click node_trainer "https://github.com/ai-14/micar-vl-moe/blob/master/train_test/caption_execute.py"
click node_captionmodel "https://github.com/ai-14/micar-vl-moe/blob/master/architecture/models.py"
click node_vision "https://github.com/ai-14/micar-vl-moe/blob/master/architecture/blocks.py"
click node_alignment "https://github.com/ai-14/micar-vl-moe/blob/master/architecture/blocks.py"
click node_moe "https://github.com/ai-14/micar-vl-moe/blob/master/architecture/blocks.py"
click node_causalmask "https://github.com/ai-14/micar-vl-moe/blob/master/architecture/utils.py"
click node_loss "https://github.com/ai-14/micar-vl-moe/blob/master/utils/losses.py"
click node_metrics "https://github.com/ai-14/micar-vl-moe/blob/master/utils/metrics.py"
click node_artifacts "https://github.com/ai-14/micar-vl-moe/blob/master/configs/cfg.py"
click node_inference "https://github.com/ai-14/micar-vl-moe/blob/master/inference.ipynb"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_covprep,node_retprep,node_grossprep,node_rocoprep,node_tokenizer,node_dataset,node_loaders toneBlue
class node_config,node_main,node_services,node_trainer,node_inference,node_datasets toneAmber
class node_captionmodel,node_vision,node_alignment,node_moe,node_causalmask toneMint
class node_loss,node_metrics,node_artifacts toneRose
class node_researcher toneIndigo
```

## Components

### Data preparation

- [COVCTR prep](https://github.com/ai-14/micar-vl-moe/blob/master/preprocessors/covctr.py): `preprocessors/covctr.py`
- [MMRetinal prep](https://github.com/ai-14/micar-vl-moe/blob/master/preprocessors/mmretinal.py): `preprocessors/mmretinal.py`
- [PGROSS prep](https://github.com/ai-14/micar-vl-moe/blob/master/preprocessors/pgross.py): `preprocessors/pgross.py`
- [ROCO prep](https://github.com/ai-14/micar-vl-moe/blob/master/preprocessors/roco.py): `preprocessors/roco.py`
- [Report tokenizer](https://github.com/ai-14/micar-vl-moe/blob/master/utils/tokenizers.py): `utils/tokenizers.py`
- [Caption dataset](https://github.com/ai-14/micar-vl-moe/blob/master/utils/datasets.py): `utils/datasets.py`
- [Data loaders](https://github.com/ai-14/micar-vl-moe/blob/master/utils/dataloaders.py): `utils/dataloaders.py`

### Experiment runtime

- [Experiment config](https://github.com/ai-14/micar-vl-moe/blob/master/configs/cfg.py): `configs/cfg.py`
- [Runtime entry](https://github.com/ai-14/micar-vl-moe/blob/master/main.py): `main.py`
- [Experiment services](https://github.com/ai-14/micar-vl-moe/blob/master/utils/services.py): `utils/services.py`
- [Train and test](https://github.com/ai-14/micar-vl-moe/blob/master/train_test/caption_execute.py): `train_test/caption_execute.py`
- [Checkpoint inference](https://github.com/ai-14/micar-vl-moe/blob/master/inference.ipynb): `inference.ipynb`

### Caption model

- [Caption model](https://github.com/ai-14/micar-vl-moe/blob/master/architecture/models.py): `architecture/models.py`
- [Multiscale vision](https://github.com/ai-14/micar-vl-moe/blob/master/architecture/blocks.py): `architecture/blocks.py`
- [Gated alignment](https://github.com/ai-14/micar-vl-moe/blob/master/architecture/blocks.py): `architecture/blocks.py`
- [MoE decoder](https://github.com/ai-14/micar-vl-moe/blob/master/architecture/blocks.py): `architecture/blocks.py`
- [Causal masking](https://github.com/ai-14/micar-vl-moe/blob/master/architecture/utils.py): `architecture/utils.py`

### Evaluation and outputs

- [Training loss](https://github.com/ai-14/micar-vl-moe/blob/master/utils/losses.py): `utils/losses.py`
- [Caption metrics](https://github.com/ai-14/micar-vl-moe/blob/master/utils/metrics.py): `utils/metrics.py`
- [Reports and checkpoints](https://github.com/ai-14/micar-vl-moe/blob/master/configs/cfg.py): `configs/cfg.py`

### Other components

- **Experimenter**
- **Medical datasets**

## Connections

- Experimenter → Runtime entry: starts run (evidence: `main.py`)
- Experimenter → COVCTR prep: runs preprocessing (evidence: `README.md`)
- Experimenter → MMRetinal prep: runs preprocessing (evidence: `README.md`)
- Experimenter → PGROSS prep: runs preprocessing (evidence: `README.md`)
- Experimenter → ROCO prep: runs preprocessing (evidence: `README.md`)
- Medical datasets → COVCTR prep: provides data (evidence: `README.md`)
- Medical datasets → MMRetinal prep: provides data (evidence: `README.md`)
- Medical datasets → PGROSS prep: provides data (evidence: `README.md`)
- Medical datasets → ROCO prep: provides data (evidence: `README.md`)
- Runtime entry → Experiment config: parses settings (evidence: `main.py`)
- Runtime entry → Experiment services: initializes run (evidence: `main.py`)
- Runtime entry → Report tokenizer: creates tokenizer (evidence: `main.py`)
- Runtime entry → Train and test: executes training (evidence: `main.py`)
- Train and test → Data loaders: builds splits (evidence: `train_test/caption_execute.py`)
- Data loaders → Caption dataset: constructs datasets (evidence: `utils/dataloaders.py`)
- Caption dataset → Report tokenizer: encodes reports (evidence: `utils/datasets.py`)
- Train and test → Caption model: trains and generates (evidence: `train_test/caption_execute.py`)
- Train and test → Training loss: computes loss (evidence: `train_test/caption_execute.py`)
- Train and test → Caption metrics: scores captions (evidence: `train_test/caption_execute.py`)
- Train and test → Experiment services: saves snapshots (evidence: `train_test/caption_execute.py`)
- Caption model → Multiscale vision: encodes images (evidence: `architecture/models.py`)
- Caption model → Gated alignment: aligns modalities (evidence: `README.md`)
- Caption model → MoE decoder: decodes captions (evidence: `README.md`)
- Caption model → Causal masking: creates masks (evidence: `architecture/models.py`)
- Train and test → Reports and checkpoints: writes results (evidence: `train_test/caption_execute.py`)
- Experimenter → Checkpoint inference: uses notebook (evidence: `README.md`)
- Checkpoint inference → Experiment services: loads checkpoint