How largeworldmodel/lwm works
Project type and purpose
• This is a research-grade multimodal machine-learning framework (not a web or mobile app). It implements a large-context autoregressive transformer (“LWM”) for language, image, and video understanding and generation, built on top of JAX (with TPU/GPU support) and PyTorch (for text inference). It provides end-to-end pipelines for data processing, model definition, training (with RingAttention and sharding), evaluation (needle/haystack benchmarks), and inference (sampling/generation/chat).
File structure
• Top level
• README.md, LICENSE, .gitignore – project metadata and setup instructions
• gpu_requirements.txt, tpu_requirements.sh – hardware-specific dependencies
• docs/ – documentation on data processing (data.md) and sharding/parallelism (sharding.md)
• imgs/ – illustrative figures used in the README/paper
• Core code (lwm/ package)
• data.py – data ingestion and preprocessing pipelines (video, text, multimodal packaging)
• llama.py – base transformer model definitions (RingAttention layers, feed-forward blocks)
• vqgan.py – VQGAN image/video encoder/decoder
• train.py – orchestrates distributed training, mesh configuration, checkpointing
• vision_chat.py, vision_generation.py, vision_llama.py – inference endpoints for vision-language chat and generation
• Scripts (scripts/)
• create_needle_data.py – builds retrieval evaluation datasets
• eval_needle*.py – single/multi‐needle benchmark evaluations
• sample_pyt.py – PyTorch sampling for text models
• shell scripts (run_*.sh) – wrapper scripts that set environment variables, locate checkpoints, and launch training/eval/inference with appropriate flags
README for additional insights
• Architecture calls out three pillars:
a. Large-context RingAttention transformer with blockwise compute (scan_attention, scan_mlp)
b. Masked sequence packing and loss weighting for mixed text/video inputs
c. Sharding/parallelism via mesh_dim (dp, fsdp, tp, sp)
• Provides end-to-end examples for:
• Language training & evaluation
• Vision-language training & chat/generation
• Details on context sizes, theta/sequence length for benchmarks
1. Data Sources
• Long-form text corpora (books)
• Video datasets
2. Data Processing Pipeline (lwm/data.py + scripts/create_needle_data.py)
• Tokenization, video frame extraction, multimodal packing, masked packing
3. Model Core (lwm/llama.py, lwm/vqgan.py)
• RingAttention transformer layers
• VQGAN encoder/decoder
• Configuration knobs (scan_attention, chunk sizes)
4. Training Orchestrator (lwm/train.py + run_train_*.sh)
• Mesh configuration (dp, fsdp, tp, sp)
• Checkpoint I/O
• Loss weighting, mixed-modality batching
5. Evaluation & Benchmarking (scripts/eval_needle*.py, run_eval_*.sh)
• Single/multi-needle retrieval tasks
6. Inference & Sampling
• Vision chat (lwm/vision_chat.py + run_vision_chat.sh)
• Vision generation (lwm/vision_generation.py + run_sample_*.sh)
• PyTorch text sampling (scripts/sample_pyt.py)
7. External Services & Storage
• Hugging Face model hub (pretrained weights)
• Checkpoint storage (GCS/S3/Local)
• TPU/GPU clusters
• Data Sources → Data Processing → Training Orchestrator → Model Checkpoints
• Checkpoints → Evaluation & Benchmarking → Metrics
• Checkpoints → Inference & Sampling → Outputs (videos, images, chat logs)
• Training Orchestrator ↔ External Resources (TPU/GPU, storage)
• Inference scripts ↔ HuggingFace for tokenizer & initial weights
c. Architectural patterns & design principles
• Modular layering: separate data, model, training, inference
• Parameter‐server‐style sharding (mesh_dim) for scaling
• Blockwise compute for near-infinite context (scan_attention/scan_mlp)
• Masked sequence packing for mixed modality input
d. Key technologies & libraries
• JAX + Pallas (TPU‐optimized) with RingAttention
• XLA backend for GPU inference
• PyTorch + Hugging Face LlamaForCausalLM (text only)
• VQGAN for vision encoding/decoding
• Shell scripts for environment/launcher abstraction
Capturing essential details
• Per‐component key parameters (e.g., mesh_dim settings, context sizes, theta values)
• Checkpoint flow is explicit: where checkpoints are written and loaded