How allenai/molmo2 works
1. What this repo is
- This is a PyTorch-based training and inference codebase for two related vision-language model families: Molmo2 and MolmoPoint.
- The architecture is centered around a shared core: dataset objects produce raw examples, model-specific preprocessors convert them into tensors, and training/evaluation code drives forward or generate.
2. Top-level execution entrypoints
- The main runnable flows are in launch_scripts/:
- pretrain.py for stage-1 pretraining
- sft.py for multitask supervised fine-tuning and long-context training
- eval.py and eval_molmo2.py for single-task and grouped evaluation
- train_gui_pointing.py for MolmoPoint-related training
- scripts/ contains operational tools: dataset download, visualization, checkpoint preparation, clip extraction, and a unified demo.
3. Model family boundary
- The shared model abstraction lives in olmo/models/model.py and model_config.py.
- Concrete model implementations are split into:
- olmo/models/molmo2/ for the main Molmo2 stack
- olmo/models/molmo_point/ for the pointing-oriented extension
- olmo/models/molmo/ for the earlier Molmo variant
- Each model package couples the network, preprocessing logic, and batching/collation behavior so the trainer can treat models uniformly.
4. Neural network building blocks
- Low-level components are in olmo/nn/.
- This includes:
- the language model wrapper (llm.py)
- image and temporal vision backbones (image_vit.py, temporal_image_vit.py, vision_backbone.py, temporal_vision_backbone.py)
- attention/distributed attention helpers (flash_attention_api.py, ulysses.py, cp_load_balancer.py)
- decoding/search support (beam_search.py)
- These modules are the main boundary between high-level multimodal model logic and raw tensor operations.
5. Data subsystem
- The dataset layer is in olmo/data/.
- It contains:
- generic dataset interfaces and loaders
- mixture/iterable composition
- packing logic
- video loading
- dataset-specific builders for benchmarks and training corpora
- The repo supports image, multi-image, video, pointing, tracking, and text-only datasets, with many benchmark-specific adapters.
6. Preprocessing pipeline
- The data path is intentionally split into stages:
- raw dataset example
- formatting into conversation/message structure
- tokenization and multimodal tensorization
- collation into a batch
- The formatting/tokenization layer lives in olmo/preprocessing/, with specialized formatters for text, image, video, points, and multiple-choice templates.
- This is the key “model-facing” boundary: datasets stay relatively raw, while preprocessors make them compatible with the target model’s prompt and tensor layout.
7. Message trees and multi-annotation examples
- The repo supports “message trees,” where one image/video can have multiple annotations.
- This is handled by message_lists in dataset examples and flattened by the multimodal preprocessor.
- A subsegment_id is added so attention masks can keep annotations separated during training or packing.
- This matters for architecture diagrams because it is a distinct control-flow branch inside preprocessing, not just a data detail.
8. Packing and long-context training
- Packing is implemented in the data pipeline and coordinated with IterableDatasetMixture.
- After preprocessing, multiple examples can be merged into a single token sequence to reduce padding and improve throughput.
- For very long sequences, the repo also supports context parallelism via olmo/nn/ulysses.py and related utilities, which shards long sequences across GPUs.
- This is especially important for video training with hundreds of frames and tens of thousands of tokens.
9. Training runtime and distributed infra
- The training loop is in olmo/train/, centered on trainer.py and run_trainer.py.
- Supporting pieces include:
- optimizer setup
- checkpointing and distributed checkpointing
- remote/shared filesystem handling
- The trainer owns logging, checkpoint lifecycle, and in-loop evaluation, and is designed for distributed torchrun-based execution.
10. Evaluation architecture
- Evaluation is a separate subsystem under olmo/eval/.
- It includes:
- task-specific evaluators
- metric computation
- model inference evaluation
- loss-based evaluation
- utilities for benchmarks such as VQA, pointing, tracking, captioning, and video understanding
- The main abstraction is a dataset/task evaluator that receives predictions plus metadata and returns benchmark metrics.
11. Hugging Face interoperability
- olmo/hf_model/ provides the conversion and HF-compatible model/processor definitions.
- This layer bridges the native checkpoint format and the Hugging Face ecosystem for:
- Transformers inference
- image/video processors
- model configuration
- checkpoint conversion
- This is a clear architectural boundary: native training checkpoints on one side, HF/vLLM consumption on the other.
12. MolmoPoint specialization
- MolmoPoint is not just a config variant; it has its own model package and preprocessing/formatted-data path.
- Key modules include a connector, text/example preprocessors, and a data formatter specialized for pointing workflows.
- In an architecture diagram, this should appear as a sibling model branch to Molmo2, sharing the training/eval infrastructure but with custom preprocessing and output formatting.
13. Operational scripts and external dependencies
- The repo relies on Hugging Face datasets/models, local or shared filesystem storage, W&B for training logs, and optionally OpenAI API keys for some evaluations.
- Scripts in scripts/ handle dataset acquisition, data inspection, checkpoint preparation, and a demo path that can expose a Gradio app.
- The README also documents cloud-scale usage assumptions like HF_DATASETS_OFFLINE, OLMO_SHARED_FS, and distributed torchrun execution.
14. Testing and validation
- tests/ covers the main boundaries: preprocessing, data iteration, point formatting, frame sampling, beam search, and optimizer behavior.
- These tests reflect the repo’s critical seams: data formatting correctness, tokenization/collation, and training-time utilities.
Main entry points: model.py Base model · preprocessing Preprocessing · trainer.py Trainer · molmo2.py Molmo2
Components 18
Execution
Entrypoints for training, evaluation, and demos.
- Launch scripts · Main runnable flows for training and evaluation.
launch_scripts - Ops scripts · Dataset, checkpoint, and demo utilities.
scripts
Models
Shared model abstractions and concrete Molmo families.
- Base model · Shared model interface and configuration boundary.
olmo/models/model.py - Molmo2 · Main Molmo2 model stack and preprocessor.
olmo/models/molmo2/molmo2.py - M2 prep · Molmo2-specific preprocessing and batching behavior.
olmo/models/molmo2/molmo2_preprocessor.py - MolmoPoint · Pointing-oriented Molmo branch with its own formatting path.
olmo/models/molmo_point/molmo_point.py - Point formatter · Pointing-specific data formatting and example preprocessing.
olmo/models/molmo_point/molmo_point_data_formatter.py
Data
Dataset loading and preprocessing pipeline.
- Data loader · Dataset loading, mixtures, packing, and iteration support.
olmo/data/data_loader.py - Datasets · Dataset adapters and corpus-specific builders.
olmo/data - Preprocessing · Text, image, video, point, and multimodal preprocessing stages.
olmo/preprocessing - Collator · Turns processed examples into model-ready batches.
olmo/preprocessing/multimodal_collator.py
Infra
Training runtime, evaluation, and HF interoperability.
- Trainer · Training loop, checkpoints, and optimization control.
olmo/train/trainer.py - Run trainer · Launcher and process orchestration for training.
olmo/train/run_trainer.py - Checkpointing · Checkpoint save/load and lifecycle management.
olmo/train/checkpointer.py - Evaluators · Task evaluators and metric aggregation.
olmo/eval/evaluators.py - Model eval · Inference and loss-based evaluation entrypoints.
olmo/eval/model_evaluator.py - HF bridge · Hugging Face configs, processors, and model conversion.
olmo/hf_model - NN core · Language model wrapper, vision backbones, attention helpers, and decoding.
olmo/nn
Connections 21
- Launch scripts → Run trainer: start training · Training scripts invoke the trainer runtime.
- Launch scripts → Model eval: run eval · Evaluation entrypoints drive model evaluation.
- Ops scripts → HF bridge: convert/prepare · Operational scripts prepare checkpoints and demos for HF use.
- Run trainer → Trainer: executes · Launcher hands control to the core trainer.
- Trainer → Checkpointing: saves state · Trainer coordinates checkpoint lifecycle.
- Trainer → Base model: trains · Trainer works through the shared model abstraction.
- Trainer → Evaluators: metrics · Trainer can run evaluation during training.
- Model eval → Evaluators: uses · Evaluation runner delegates to task evaluators.
- Evaluators → Base model: queries model · Evaluators call into model inference/loss APIs.
- Base model → Molmo2: implements · Molmo2 is the main concrete model family.
- Base model → MolmoPoint: implements · MolmoPoint is a sibling concrete model family.
- Molmo2 → M2 prep: preprocess · Molmo2 bundles its model with model-specific preprocessing.
- MolmoPoint → Point formatter: formats · MolmoPoint uses specialized pointing-oriented formatting.
- Datasets → Data loader: feeds · Dataset builders are consumed by the loader and mixture layer.
- Data loader → Preprocessing: raw examples · Loaded examples flow into preprocessing.
- Preprocessing → Collator: batch · Processed tensors are collated into batches.
- Preprocessing → M2 prep: m2 path · Molmo2 preprocessing consumes the shared preprocessing stages.
- Preprocessing → Point formatter: point path · MolmoPoint uses its own data formatting branch.
- Molmo2 → NN core: uses · Molmo2 is built on shared neural network components.
- MolmoPoint → NN core: uses · MolmoPoint also depends on the shared tensor ops stack.
- HF bridge → Base model: bridges · HF conversion mirrors the native model boundary.