# allenai/molmo2 architecture

> Architecture diagram of the GitHub repository allenai/molmo2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/allenai/molmo2
- Repository: https://github.com/allenai/molmo2

## Overview

1. **What this repo is**
   - This is a PyTorch-based training and inference codebase for two related vision-language model families: **Molmo2** and **MolmoPoint**.
   - The architecture is centered around a shared core: dataset objects produce raw examples, model-specific preprocessors convert them into tensors, and training/evaluation code drives `forward` or `generate`.

2. **Top-level execution entrypoints**
   - The main runnable flows are in `launch_scripts/`:
     - `pretrain.py` for stage-1 pretraining
     - `sft.py` for multitask supervised fine-tuning and long-context training
     - `eval.py` and `eval_molmo2.py` for single-task and grouped evaluation
     - `train_gui_pointing.py` for MolmoPoint-related training
   - `scripts/` contains operational tools: dataset download, visualization, checkpoint preparation, clip extraction, and a unified demo.

3. **Model family boundary**
   - The shared model abstraction lives in `olmo/models/model.py` and `model_config.py`.
   - Concrete model implementations are split into:
     - `olmo/models/molmo2/` for the main Molmo2 stack
     - `olmo/models/molmo_point/` for the pointing-oriented extension
     - `olmo/models/molmo/` for the earlier Molmo variant
   - Each model package couples the network, preprocessing logic, and batching/collation behavior so the trainer can treat models uniformly.

4. **Neural network building blocks**
   - Low-level components are in `olmo/nn/`.
   - This includes:
     - the language model wrapper (`llm.py`)
     - image and temporal vision backbones (`image_vit.py`, `temporal_image_vit.py`, `vision_backbone.py`, `temporal_vision_backbone.py`)
     - attention/distributed attention helpers (`flash_attention_api.py`, `ulysses.py`, `cp_load_balancer.py`)
     - decoding/search support (`beam_search.py`)
   - These modules are the main boundary between high-level multimodal model logic and raw tensor operations.

5. **Data subsystem**
   - The dataset layer is in `olmo/data/`.
   - It contains:
     - generic dataset interfaces and loaders
     - mixture/iterable composition
     - packing logic
     - video loading
     - dataset-specific builders for benchmarks and training corpora
   - The repo supports image, multi-image, video, pointing, tracking, and text-only datasets, with many benchmark-specific adapters.

6. **Preprocessing pipeline**
   - The data path is intentionally split into stages:
     - raw dataset example
     - formatting into conversation/message structure
     - tokenization and multimodal tensorization
     - collation into a batch
   - The formatting/tokenization layer lives in `olmo/preprocessing/`, with specialized formatters for text, image, video, points, and multiple-choice templates.
   - This is the key “model-facing” boundary: datasets stay relatively raw, while preprocessors make them compatible with the target model’s prompt and tensor layout.

7. **Message trees and multi-annotation examples**
   - The repo supports “message trees,” where one image/video can have multiple annotations.
   - This is handled by `message_lists` in dataset examples and flattened by the multimodal preprocessor.
   - A `subsegment_id` is added so attention masks can keep annotations separated during training or packing.
   - This matters for architecture diagrams because it is a distinct control-flow branch inside preprocessing, not just a data detail.

8. **Packing and long-context training**
   - Packing is implemented in the data pipeline and coordinated with `IterableDatasetMixture`.
   - After preprocessing, multiple examples can be merged into a single token sequence to reduce padding and improve throughput.
   - For very long sequences, the repo also supports **context parallelism** via `olmo/nn/ulysses.py` and related utilities, which shards long sequences across GPUs.
   - This is especially important for video training with hundreds of frames and tens of thousands of tokens.

9. **Training runtime and distributed infra**
   - The training loop is in `olmo/train/`, centered on `trainer.py` and `run_trainer.py`.
   - Supporting pieces include:
     - optimizer setup
     - checkpointing and distributed checkpointing
     - remote/shared filesystem handling
   - The trainer owns logging, checkpoint lifecycle, and in-loop evaluation, and is designed for distributed torchrun-based execution.

10. **Evaluation architecture**
    - Evaluation is a separate subsystem under `olmo/eval/`.
    - It includes:
      - task-specific evaluators
      - metric computation
      - model inference evaluation
      - loss-based evaluation
      - utilities for benchmarks such as VQA, pointing, tracking, captioning, and video understanding
    - The main abstraction is a dataset/task evaluator that receives predictions plus metadata and returns benchmark metrics.

11. **Hugging Face interoperability**
    - `olmo/hf_model/` provides the conversion and HF-compatible model/processor definitions.
    - This layer bridges the native checkpoint format and the Hugging Face ecosystem for:
      - Transformers inference
      - image/video processors
      - model configuration
      - checkpoint conversion
    - This is a clear architectural boundary: native training checkpoints on one side, HF/vLLM consumption on the other.

12. **MolmoPoint specialization**
    - MolmoPoint is not just a config variant; it has its own model package and preprocessing/formatted-data path.
    - Key modules include a connector, text/example preprocessors, and a data formatter specialized for pointing workflows.
    - In an architecture diagram, this should appear as a sibling model branch to Molmo2, sharing the training/eval infrastructure but with custom preprocessing and output formatting.

13. **Operational scripts and external dependencies**
    - The repo relies on Hugging Face datasets/models, local or shared filesystem storage, W&B for training logs, and optionally OpenAI API keys for some evaluations.
    - Scripts in `scripts/` handle dataset acquisition, data inspection, checkpoint preparation, and a demo path that can expose a Gradio app.
    - The README also documents cloud-scale usage assumptions like `HF_DATASETS_OFFLINE`, `OLMO_SHARED_FS`, and distributed torchrun execution.

14. **Testing and validation**
    - `tests/` covers the main boundaries: preprocessing, data iteration, point formatting, frame sampling, beam search, and optimizer behavior.
    - These tests reflect the repo’s critical seams: data formatting correctness, tokenization/collation, and training-time utilities.

Main entry points: [`olmo/models/model.py`](https://github.com/allenai/molmo2/blob/main/olmo/models/model.py) (Base model), [`olmo/preprocessing`](https://github.com/allenai/molmo2/tree/main/olmo/preprocessing) (Preprocessing), [`olmo/train/trainer.py`](https://github.com/allenai/molmo2/blob/main/olmo/train/trainer.py) (Trainer), [`olmo/models/molmo2/molmo2.py`](https://github.com/allenai/molmo2/blob/main/olmo/models/molmo2/molmo2.py) (Molmo2)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_exec["Execution"]
  node_launch_scripts["Launch scripts<br/>entrypoints"]
  node_scripts["Ops scripts<br/>ops tools"]
end

subgraph group_models["Models"]
  node_model_base["Base model<br/>model abstraction<br/>[model.py]"]
  node_molmo2["Molmo2<br/>vision-language model<br/>[molmo2.py]"]
  node_molmo2_prep["M2 prep<br/>multimodal preprocessing"]
  node_molmo_point["MolmoPoint<br/>pointing model<br/>[molmo_point.py]"]
  node_mp_formatter["Point formatter<br/>data formatting"]
end

subgraph group_data["Data"]
  node_data_loader["Data loader<br/>dataset runtime<br/>[data_loader.py]"]
  node_datasets["Datasets<br/>dataset builders"]
  node_preprocessing["Preprocessing<br/>tokenization/tensorization"]
  node_collator["Collator<br/>batching"]
end

subgraph group_infra["Infra"]
  node_trainer["Trainer<br/>distributed training<br/>[trainer.py]"]
  node_run_trainer["Run trainer<br/>trainer entry<br/>[run_trainer.py]"]
  node_checkpointer["Checkpointing<br/>ckpt mgmt<br/>[checkpointer.py]"]
  node_evaluators["Evaluators<br/>evaluation core<br/>[evaluators.py]"]
  node_model_eval["Model eval<br/>eval runner<br/>[model_evaluator.py]"]
  node_hf_model["HF bridge<br/>hf interoperability"]
  node_nn_core["NN core<br/>tensor ops"]
end

node_launch_scripts -->|"start training"| node_run_trainer
node_launch_scripts -->|"run eval"| node_model_eval
node_scripts -->|"convert/prepare"| node_hf_model
node_run_trainer -->|"executes"| node_trainer
node_trainer -->|"saves state"| node_checkpointer
node_trainer -->|"trains"| node_model_base
node_trainer -->|"metrics"| node_evaluators
node_model_eval -->|"uses"| node_evaluators
node_evaluators -->|"queries model"| node_model_base
node_model_base -->|"implements"| node_molmo2
node_model_base -->|"implements"| node_molmo_point
node_molmo2 -->|"preprocess"| node_molmo2_prep
node_molmo_point -->|"formats"| node_mp_formatter
node_datasets -->|"feeds"| node_data_loader
node_data_loader -->|"raw examples"| node_preprocessing
node_preprocessing -->|"batch"| node_collator
node_preprocessing -->|"m2 path"| node_molmo2_prep
node_preprocessing -->|"point path"| node_mp_formatter
node_molmo2 -->|"uses"| node_nn_core
node_molmo_point -->|"uses"| node_nn_core
node_hf_model -->|"bridges"| node_model_base

click node_launch_scripts "https://github.com/allenai/molmo2/tree/main/launch_scripts"
click node_scripts "https://github.com/allenai/molmo2/tree/main/scripts"
click node_trainer "https://github.com/allenai/molmo2/blob/main/olmo/train/trainer.py"
click node_run_trainer "https://github.com/allenai/molmo2/blob/main/olmo/train/run_trainer.py"
click node_checkpointer "https://github.com/allenai/molmo2/blob/main/olmo/train/checkpointer.py"
click node_evaluators "https://github.com/allenai/molmo2/blob/main/olmo/eval/evaluators.py"
click node_model_eval "https://github.com/allenai/molmo2/blob/main/olmo/eval/model_evaluator.py"
click node_hf_model "https://github.com/allenai/molmo2/tree/main/olmo/hf_model"
click node_model_base "https://github.com/allenai/molmo2/blob/main/olmo/models/model.py"
click node_molmo2 "https://github.com/allenai/molmo2/blob/main/olmo/models/molmo2/molmo2.py"
click node_molmo2_prep "https://github.com/allenai/molmo2/blob/main/olmo/models/molmo2/molmo2_preprocessor.py"
click node_molmo_point "https://github.com/allenai/molmo2/blob/main/olmo/models/molmo_point/molmo_point.py"
click node_mp_formatter "https://github.com/allenai/molmo2/blob/main/olmo/models/molmo_point/molmo_point_data_formatter.py"
click node_data_loader "https://github.com/allenai/molmo2/blob/main/olmo/data/data_loader.py"
click node_datasets "https://github.com/allenai/molmo2/tree/main/olmo/data"
click node_preprocessing "https://github.com/allenai/molmo2/tree/main/olmo/preprocessing"
click node_collator "https://github.com/allenai/molmo2/blob/main/olmo/preprocessing/multimodal_collator.py"
click node_nn_core "https://github.com/allenai/molmo2/tree/main/olmo/nn"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_launch_scripts,node_scripts toneBlue
class node_model_base,node_molmo2,node_molmo2_prep,node_molmo_point,node_mp_formatter toneAmber
class node_data_loader,node_datasets,node_preprocessing,node_collator toneMint
class node_trainer,node_run_trainer,node_checkpointer,node_evaluators,node_model_eval,node_hf_model,node_nn_core toneRose
```

## Components

### Execution

Entrypoints for training, evaluation, and demos.

- [Launch scripts](https://github.com/allenai/molmo2/tree/main/launch_scripts): entrypoints · Main runnable flows for training and evaluation. · `launch_scripts`
- [Ops scripts](https://github.com/allenai/molmo2/tree/main/scripts): ops tools · Dataset, checkpoint, and demo utilities. · `scripts`

### Models

Shared model abstractions and concrete Molmo families.

- [Base model](https://github.com/allenai/molmo2/blob/main/olmo/models/model.py): model abstraction · Shared model interface and configuration boundary. · `olmo/models/model.py`
- [Molmo2](https://github.com/allenai/molmo2/blob/main/olmo/models/molmo2/molmo2.py): vision-language model · Main Molmo2 model stack and preprocessor. · `olmo/models/molmo2/molmo2.py`
- [M2 prep](https://github.com/allenai/molmo2/blob/main/olmo/models/molmo2/molmo2_preprocessor.py): multimodal preprocessing · Molmo2-specific preprocessing and batching behavior. · `olmo/models/molmo2/molmo2_preprocessor.py`
- [MolmoPoint](https://github.com/allenai/molmo2/blob/main/olmo/models/molmo_point/molmo_point.py): pointing model · Pointing-oriented Molmo branch with its own formatting path. · `olmo/models/molmo_point/molmo_point.py`
- [Point formatter](https://github.com/allenai/molmo2/blob/main/olmo/models/molmo_point/molmo_point_data_formatter.py): data formatting · Pointing-specific data formatting and example preprocessing. · `olmo/models/molmo_point/molmo_point_data_formatter.py`

### Data

Dataset loading and preprocessing pipeline.

- [Data loader](https://github.com/allenai/molmo2/blob/main/olmo/data/data_loader.py): dataset runtime · Dataset loading, mixtures, packing, and iteration support. · `olmo/data/data_loader.py`
- [Datasets](https://github.com/allenai/molmo2/tree/main/olmo/data): dataset builders · Dataset adapters and corpus-specific builders. · `olmo/data`
- [Preprocessing](https://github.com/allenai/molmo2/tree/main/olmo/preprocessing): tokenization/tensorization · Text, image, video, point, and multimodal preprocessing stages. · `olmo/preprocessing`
- [Collator](https://github.com/allenai/molmo2/blob/main/olmo/preprocessing/multimodal_collator.py): batching · Turns processed examples into model-ready batches. · `olmo/preprocessing/multimodal_collator.py`

### Infra

Training runtime, evaluation, and HF interoperability.

- [Trainer](https://github.com/allenai/molmo2/blob/main/olmo/train/trainer.py): distributed training · Training loop, checkpoints, and optimization control. · `olmo/train/trainer.py`
- [Run trainer](https://github.com/allenai/molmo2/blob/main/olmo/train/run_trainer.py): trainer entry · Launcher and process orchestration for training. · `olmo/train/run_trainer.py`
- [Checkpointing](https://github.com/allenai/molmo2/blob/main/olmo/train/checkpointer.py): ckpt mgmt · Checkpoint save/load and lifecycle management. · `olmo/train/checkpointer.py`
- [Evaluators](https://github.com/allenai/molmo2/blob/main/olmo/eval/evaluators.py): evaluation core · Task evaluators and metric aggregation. · `olmo/eval/evaluators.py`
- [Model eval](https://github.com/allenai/molmo2/blob/main/olmo/eval/model_evaluator.py): eval runner · Inference and loss-based evaluation entrypoints. · `olmo/eval/model_evaluator.py`
- [HF bridge](https://github.com/allenai/molmo2/tree/main/olmo/hf_model): hf interoperability · Hugging Face configs, processors, and model conversion. · `olmo/hf_model`
- [NN core](https://github.com/allenai/molmo2/tree/main/olmo/nn): tensor ops · Language model wrapper, vision backbones, attention helpers, and decoding. · `olmo/nn`

## Connections

- Launch scripts → Run trainer: start training · Training scripts invoke the trainer runtime.
- Launch scripts → Model eval: run eval · Evaluation entrypoints drive model evaluation.
- Ops scripts → HF bridge: convert/prepare · Operational scripts prepare checkpoints and demos for HF use.
- Run trainer → Trainer: executes · Launcher hands control to the core trainer.
- Trainer → Checkpointing: saves state · Trainer coordinates checkpoint lifecycle.
- Trainer → Base model: trains · Trainer works through the shared model abstraction.
- Trainer → Evaluators: metrics · Trainer can run evaluation during training.
- Model eval → Evaluators: uses · Evaluation runner delegates to task evaluators.
- Evaluators → Base model: queries model · Evaluators call into model inference/loss APIs.
- Base model → Molmo2: implements · Molmo2 is the main concrete model family.
- Base model → MolmoPoint: implements · MolmoPoint is a sibling concrete model family.
- Molmo2 → M2 prep: preprocess · Molmo2 bundles its model with model-specific preprocessing.
- MolmoPoint → Point formatter: formats · MolmoPoint uses specialized pointing-oriented formatting.
- Datasets → Data loader: feeds · Dataset builders are consumed by the loader and mixture layer.
- Data loader → Preprocessing: raw examples · Loaded examples flow into preprocessing.
- Preprocessing → Collator: batch · Processed tensors are collated into batches.
- Preprocessing → M2 prep: m2 path · Molmo2 preprocessing consumes the shared preprocessing stages.
- Preprocessing → Point formatter: point path · MolmoPoint uses its own data formatting branch.
- Molmo2 → NN core: uses · Molmo2 is built on shared neural network components.
- MolmoPoint → NN core: uses · MolmoPoint also depends on the shared tensor ops stack.
- HF bridge → Base model: bridges · HF conversion mirrors the native model boundary.
