# allenai/olmo architecture

> Architecture diagram of the GitHub repository allenai/olmo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/allenai/olmo
- Repository: https://github.com/allenai/olmo

## Overview

##### Main Components
- Data Ingestion & Preprocessing:
– This includes directories like “olmo_data” (with dataset files, streaming configurations, and support for Hugging Face datasets) and parts of “olmo/data”.
– Represent the external data sources (e.g. “OLMo-mix-1124” and “Dolmino-mix-1124”) that feed into the system.
- Training Pipeline and Engine:
– Encapsulate the distributed training scripts (e.g., “scripts/train.py”, usage of torchrun, and configuration files in “configs/official-1124”, “configs/tiny”, etc.).
– Make sure to note the two-stage training process (Stage 1 on massive web data followed by Stage 2 fine-tuning on high-quality data with multiple seeds and subsequent weight averaging).
– Include the core model definition and optimizer functions found in “olmo/model.py”, “olmo/optim.py”, “olmo/train.py”, etc.
- Model Conversion and Quantization:
– Highlight “hf_olmo” and parts of “inference/compression”.
– This module converts internal OLMo checkpoint formats to Hugging Face–compatible formats and supports quantization (8-bit inference, safe tensors conversion, etc.).
- Evaluation and Testing:
– Represent the “evaluation” folder and testing modules such as “tests/eval”, “tests/model_test.py”, “tests/data/…”.
– This component includes evaluation configurations (pipelined tasks, perplexity tests, downstream task evaluations) and uses external evaluation repositories (like OLMo-eval).
- Inference/Deployment:
– Capture the “inference” directory and related scripts that integrate with Hugging Face transformers.
– Include deployment modules such as the Modal.com hosting script found in “scripts/olmo2_modal_openai.py”.
– Represent how the inference component accepts tokenized inputs (via AutoTokenizer) and returns generated text.
- Utility & Configuration:
– List the multitude of configuration files (YAML, CSV) in the “configs” directory that define model sizes, training hyperparameters, and specific annealing configurations.
– Include various utility scripts under “scripts” that support conversion, changelog preparation, data sharding, and performance monitoring.

##### Relationships & Interactions
- Data flows from external datasets into the Data Ingestion module, where data is either streamed live (via HTTP as indicated in the README) or downloaded for local processing.
- The processed data is fed into the distributed Training Pipeline where Torch-based training (via torchrun) uses the provided YAML configuration files.
- Checkpoints produced during Stage 1 serve as the starting point for Stage 2; several training jobs (with different random seeds and token counts) run in parallel, and their weights are later “souped” (averaged) together.
- Trained models are then passed through Model Conversion scripts (in “hf_olmo” and “inference/compression”) to be adapted for inference, quantization, or further fine-tuning.
- The Evaluation module then uses these converted models to run downstream and language modeling tasks that verify performance.
- Finally, the Inference module (integrated with Hugging Face’s pipeline API, as described in the README) and deployment scripts provide interfaces for end users to query the model.

##### Architectural Patterns & Design Principles
- Modularity: Each phase (data, training, conversion, evaluation, inference) is decoupled into its folder or micro–module, making it easier to update, test, or replace individual components.
- Distributed / Scalable Training: The system leverages torchrun for multi-GPU training and has configurations for multiple stages and seeds.
- Hybrid Deployment: Integration with both on-premise (via docker, scripts, and PyTorch) and cloud-based (Modal.com hosting) environments.
- Configuration-Driven: The use of YAML/CSV configuration files in “configs” allows dynamic tuning of hyperparameters and easy reproduction of training runs.
- Extensibility: The conversion scripts and evaluation tests provide hooks for integrating new data sources, models, or processing techniques.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Sources
    ES1("OLMo-mix-1124"):::external
    ES2("Dolmino-mix-1124"):::external

    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        DI1("Streaming & HF Datasets"):::data
        DI2("Local Data Files"):::data
        DI_Out("Processed Data"):::data
        DI1 --> DI_Out
        DI2 --> DI_Out
    end

    %% Training Pipeline and Engine
    subgraph "Training Pipeline and Engine"
        TS("Distributed Training (torchrun)"):::training
        T2("OLMo Training Module"):::training
        MODEL("Model Definition"):::training
        OPTIM("Optimizer Functions"):::training
        CONFIG("Config Files (YAML/CSV)"):::utility
        STAGE("Two-Stage Training Process"):::training
        CHECKPOINT("OLMo Checkpoint"):::training
        TS --> T2
        T2 --> STAGE
        STAGE --> CHECKPOINT
        MODEL --- TS
        OPTIM --- TS
        CONFIG --- TS
    end

    %% Model Conversion and Quantization
    subgraph "Model Conversion & Quantization"
        CONV_A("Conversion Script A"):::conversion
        CONV_B("Conversion Script B"):::conversion
        QUANT("Quantization & Compression"):::conversion
        SAFE("Safetensors Conversion"):::conversion
        FINAL_CONV("Converted Model"):::conversion
        CHECKPOINT --> CONV_A
        CONV_A --> CONV_B
        CONV_B --> QUANT
        QUANT --> SAFE
        SAFE --> FINAL_CONV
    end

    %% Evaluation and Testing
    subgraph "Evaluation & Testing"
        EVAL("Evaluation Module"):::evaluation
    end

    %% Inference and Deployment
    subgraph "Inference & Deployment"
        INF("Hugging Face Inference Integration"):::inference
    end

    %% Utility & Configuration
    subgraph "Utility & Configuration"
        UTIL("Utility & Configurations"):::utility
        UTILSCRIPTS("Utility Scripts"):::utility
    end

    %% Connections
    ES1 --> DI1
    ES2 --> DI1
    DI_Out --> TS
    FINAL_CONV --> EVAL
    FINAL_CONV --> INF
    UTIL --- TS
    UTILSCRIPTS --- INF

    %% Click Events for Data Ingestion & Preprocessing
    click DI1 "https://github.com/allenai/olmo/tree/main/olmo_data"
    click DI2 "https://github.com/allenai/olmo/tree/main/olmo/data"

    %% Click Events for Training Pipeline and Engine
    click TS "https://github.com/allenai/olmo/blob/main/scripts/train.py"
    click T2 "https://github.com/allenai/olmo/blob/main/olmo/train.py"
    click CONFIG "https://github.com/allenai/olmo/tree/main/configs/official-1124"
    click MODEL "https://github.com/allenai/olmo/blob/main/olmo/model.py"
    click OPTIM "https://github.com/allenai/olmo/blob/main/olmo/optim.py"

    %% Click Events for Model Conversion and Quantization
    click CONV_A "https://github.com/allenai/olmo/blob/main/scripts/convert_olmo2_to_hf.py"
    click CONV_A "https://github.com/allenai/olmo/tree/main/hf_olmo"
    click CONV_B "https://github.com/allenai/olmo/blob/main/scripts/convert_olmo_to_hf_new.py"
    click QUANT "https://github.com/allenai/olmo/tree/main/inference/compression"
    click SAFE "https://github.com/allenai/olmo/blob/main/scripts/convert_pt_to_safetensors.py"

    %% Click Events for Evaluation and Testing
    click EVAL "https://github.com/allenai/olmo/tree/main/evaluation"
    click EVAL "https://github.com/allenai/olmo/tree/main/tests/eval"
    click EVAL "https://github.com/allenai/olmo/blob/main/tests/model_test.py"

    %% Click Events for Inference and Deployment
    click INF "https://github.com/allenai/olmo/tree/main/inference"
    click INF "https://github.com/allenai/olmo/blob/main/scripts/olmo2_modal_openai.py"

    %% Click Events for Utility & Configuration
    click UTIL "https://github.com/allenai/olmo/tree/main/configs"
    click UTILSCRIPTS "https://github.com/allenai/olmo/blob/main/scripts/prepare_changelog.py"
    click UTILSCRIPTS "https://github.com/allenai/olmo/blob/main/scripts/s3_unshard_to_hf.py"

    %% Styles
    classDef data fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef training fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef conversion fill:#98FB98,stroke:#000,stroke-width:2px;
    classDef evaluation fill:#DDA0DD,stroke:#000,stroke-width:2px;
    classDef inference fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef utility fill:#FFA07A,stroke:#000,stroke-width:2px;
    classDef external fill:#E6E6FA,stroke:#000,stroke-width:2px;
```
