# largeworldmodel/lwm architecture

> Architecture diagram of the GitHub repository largeworldmodel/lwm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/largeworldmodel/lwm
- Repository: https://github.com/largeworldmodel/lwm

## Overview

##### Project type and purpose
- This is a research-grade multimodal machine-learning framework (not a web or mobile app). It implements a large-context autoregressive transformer (“LWM”) for language, image, and video understanding and generation, built on top of JAX (with TPU/GPU support) and PyTorch (for text inference). It provides end-to-end pipelines for data processing, model definition, training (with RingAttention and sharding), evaluation (needle/haystack benchmarks), and inference (sampling/generation/chat).

##### File structure
- Top level
- README.md, LICENSE, .gitignore – project metadata and setup instructions
- gpu_requirements.txt, tpu_requirements.sh – hardware-specific dependencies
- docs/ – documentation on data processing (data.md) and sharding/parallelism (sharding.md)
- imgs/ – illustrative figures used in the README/paper
- Core code (`lwm/` package)
- data.py – data ingestion and preprocessing pipelines (video, text, multimodal packaging)
- llama.py – base transformer model definitions (RingAttention layers, feed-forward blocks)
- vqgan.py – VQGAN image/video encoder/decoder
- train.py – orchestrates distributed training, mesh configuration, checkpointing
- vision_chat.py, vision_generation.py, vision_llama.py – inference endpoints for vision-language chat and generation
- Scripts (`scripts/`)
- create_needle_data.py – builds retrieval evaluation datasets
- eval_needle*.py – single/multi‐needle benchmark evaluations
- sample_pyt.py – PyTorch sampling for text models
- shell scripts (`run_*.sh`) – wrapper scripts that set environment variables, locate checkpoints, and launch training/eval/inference with appropriate flags

##### README for additional insights
- Architecture calls out three pillars:
a. Large-context RingAttention transformer with blockwise compute (scan_attention, scan_mlp)
b. Masked sequence packing and loss weighting for mixed text/video inputs
c. Sharding/parallelism via `mesh_dim` (dp, fsdp, tp, sp)
- Provides end-to-end examples for:
- Language training & evaluation
- Vision-language training & chat/generation
- Details on context sizes, `theta`/sequence length for benchmarks

1. Data Sources
- Long-form text corpora (books)
- Video datasets
2. Data Processing Pipeline (`lwm/data.py` + `scripts/create_needle_data.py`)
- Tokenization, video frame extraction, multimodal packing, masked packing
3. Model Core (`lwm/llama.py`, `lwm/vqgan.py`)
- RingAttention transformer layers
- VQGAN encoder/decoder
- Configuration knobs (scan_attention, chunk sizes)
4. Training Orchestrator (`lwm/train.py` + `run_train_*.sh`)
- Mesh configuration (dp, fsdp, tp, sp)
- Checkpoint I/O
- Loss weighting, mixed-modality batching
5. Evaluation & Benchmarking (`scripts/eval_needle*.py`, `run_eval_*.sh`)
- Single/multi-needle retrieval tasks
6. Inference & Sampling
- Vision chat (`lwm/vision_chat.py` + `run_vision_chat.sh`)
- Vision generation (`lwm/vision_generation.py` + `run_sample_*.sh`)
- PyTorch text sampling (`scripts/sample_pyt.py`)
7. External Services & Storage
- Hugging Face model hub (pretrained weights)
- Checkpoint storage (GCS/S3/Local)
- TPU/GPU clusters

- Data Sources → Data Processing → Training Orchestrator → Model Checkpoints
- Checkpoints → Evaluation & Benchmarking → Metrics
- Checkpoints → Inference & Sampling → Outputs (videos, images, chat logs)
- Training Orchestrator ↔ External Resources (TPU/GPU, storage)
- Inference scripts ↔ HuggingFace for tokenizer & initial weights

c. Architectural patterns & design principles
- Modular layering: separate data, model, training, inference
- Parameter‐server‐style sharding (mesh_dim) for scaling
- Blockwise compute for near-infinite context (scan_attention/scan_mlp)
- Masked sequence packing for mixed modality input

d. Key technologies & libraries
- JAX + Pallas (TPU‐optimized) with RingAttention
- XLA backend for GPU inference
- PyTorch + Hugging Face `LlamaForCausalLM` (text only)
- VQGAN for vision encoding/decoding
- Shell scripts for environment/launcher abstraction

##### Capturing essential details
- Per‐component key parameters (e.g., mesh_dim settings, context sizes, theta values)
- Checkpoint flow is explicit: where checkpoints are written and loaded

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    subgraph "Data Sources"
        TextCorpus["Long-form Text Corpora"]:::dataPipeline
        VideoDataset["Video Datasets"]:::dataPipeline
    end

    %% Data Processing
    subgraph "Data Processing"
        DataPipeline["Data Processing Pipeline"]:::dataPipeline
        ScriptsProcessing[/Data Processing Scripts/]:::dataPipeline
    end

    TextCorpus -->|feeds| DataPipeline
    VideoDataset -->|feeds| DataPipeline
    ScriptsProcessing -->|builds evaluation data| DataPipeline

    %% Model Core
    subgraph "Model Core"
        Transformer["Transformer Definition"]:::model
        VQGAN["VQGAN Encoder/Decoder"]:::model
    end

    %% Training Orchestrator
    subgraph "Training Orchestrator"
        TrainOrch["Training Orchestrator"]:::training
        TextTrainScript[/Text-only Training Script/]:::training
        VisionTrainScript[/Vision-Language Training Script/]:::training
    end

    DataPipeline -->|preprocessed data| TrainOrch
    Transformer --> TrainOrch
    VQGAN --> TrainOrch

    %% Evaluation & Benchmarking
    subgraph "Evaluation & Benchmarking"
        EvalSingle["Single-Needle Evaluation"]:::inference
        EvalMulti["Multi-Needle Evaluation"]:::inference
        SingleLaunch[/Single-Needle Launch Script/]:::inference
        MultiLaunch[/Multi-Needle Launch Script/]:::inference
    end

    %% Inference & Sampling
    subgraph "Inference & Sampling"
        VisionChat["Vision Chat"]:::inference
        VisionGen["Vision Generation"]:::inference
        TextSample["Text Sampling"]:::inference
        VisionLlama["Vision+Llama Bridge"]:::inference
        ChatLaunch[/Vision Chat Launch Script/]:::inference
        GenImageLaunch[/Image Sampling Launch Script/]:::inference
        GenVideoLaunch[/Video Sampling Launch Script/]:::inference
        TextSampleScript[/PyTorch Text Sampler/]:::inference
    end

    %% External Dependencies
    subgraph "External Dependencies"
        HF["Hugging Face Model Hub"]:::external
        Storage["Checkpoint Storage (GCS/S3/Local)"]:::external
        Compute["TPU/GPU Clusters"]:::external
    end

    TrainOrch -->|writes checkpoints| Storage
    TrainOrch -->|computes on| Compute
    Storage -->|loads for| EvalSingle
    Storage -->|loads for| EvalMulti
    Storage -->|loads for| VisionChat
    Storage -->|loads for| VisionGen
    Storage -->|loads for| TextSample
    HF -->|provides tokenizer/weights| VisionChat
    HF -->|provides tokenizer/weights| VisionGen
    HF -->|provides tokenizer/weights| TextSample

    EvalSingle --> SingleLaunch
    EvalMulti --> MultiLaunch
    VisionChat --> ChatLaunch
    VisionGen --> GenImageLaunch
    VisionGen --> GenVideoLaunch
    TextSample --> TextSampleScript

    %% Click Events
    click DataPipeline "https://github.com/largeworldmodel/lwm/blob/main/lwm/data.py"
    click ScriptsProcessing "https://github.com/largeworldmodel/lwm/blob/main/scripts/create_needle_data.py"
    click Transformer "https://github.com/largeworldmodel/lwm/blob/main/lwm/llama.py"
    click VQGAN "https://github.com/largeworldmodel/lwm/blob/main/lwm/vqgan.py"
    click TrainOrch "https://github.com/largeworldmodel/lwm/blob/main/lwm/train.py"
    click TextTrainScript "https://github.com/largeworldmodel/lwm/blob/main/scripts/run_train_text.sh"
    click VisionTrainScript "https://github.com/largeworldmodel/lwm/blob/main/scripts/run_train_vision_text.sh"
    click EvalSingle "https://github.com/largeworldmodel/lwm/blob/main/scripts/eval_needle.py"
    click EvalMulti "https://github.com/largeworldmodel/lwm/blob/main/scripts/eval_needle_multi.py"
    click SingleLaunch "https://github.com/largeworldmodel/lwm/blob/main/scripts/run_eval_needle.sh"
    click MultiLaunch "https://github.com/largeworldmodel/lwm/blob/main/scripts/run_eval_needle_multi.sh"
    click VisionChat "https://github.com/largeworldmodel/lwm/blob/main/lwm/vision_chat.py"
    click ChatLaunch "https://github.com/largeworldmodel/lwm/blob/main/scripts/run_vision_chat.sh"
    click VisionGen "https://github.com/largeworldmodel/lwm/blob/main/lwm/vision_generation.py"
    click GenImageLaunch "https://github.com/largeworldmodel/lwm/blob/main/scripts/run_sample_image.sh"
    click GenVideoLaunch "https://github.com/largeworldmodel/lwm/blob/main/scripts/run_sample_video.sh"
    click TextSampleScript "https://github.com/largeworldmodel/lwm/blob/main/scripts/sample_pyt.py"
    click VisionLlama "https://github.com/largeworldmodel/lwm/blob/main/lwm/vision_llama.py"

    %% Styles
    classDef dataPipeline fill:#D0E7FF,stroke:#666,stroke-width:1px
    classDef model fill:#D0FFDC,stroke:#666,stroke-width:1px
    classDef training fill:#FFE0B2,stroke:#666,stroke-width:1px
    classDef inference fill:#E1BEE7,stroke:#666,stroke-width:1px
    classDef external fill:#EEEEEE,stroke:#666,stroke-width:1px
```
