How ai-hypercomputer/torchprime works
Project Type & Purpose
• torchprime is an open-source reference implementation and CLI toolset for high-performance PyTorch model training on Google Cloud TPUs. It provides:
• A CLI/launcher (torchprime/launcher) for local and multi-VM distributed jobs via XPK.
• A core training engine (torchprime/torch_xla_models) with Hydra config support, FSDP/SPMD parallelism, model definitions, and dataset integration.
• Utilities for data loading, sharding, mesh creation, profiling, metrics collection, and BigQuery reporting.
• Experimental and GPU-original model implementations for comparison.
Key Components & Structure
A. CLI & Orchestration
• Entry point: torchprime/launcher/cli.py (commands: tp, torchprime).
• Docker build & push: launcher/Dockerfile, buildpush.py.
• Remote run wrapper: tp run which uses XPK to broadcast to TPU VMs.
• Metrics upload: upload_metrics_to_bq.py.
B. Configuration Layer
• Hydra config files in torch_xla_models/configs/ and hf_models/configs/.
• Overrides via CLI key=value or --config-name.
• pyproject.toml defines package metadata and dependencies.
C. Core Training Engine
• Training script: torchprime/torch_xla_models/train.py.
• Trainer abstractions: trainer/base_trainer.py, trainer/sft_trainer.py.
• Model implementations under model/ and model_rewriting/ for Llama, Mixtral, etc.
• Data classes: torchprime/data/dataset.py, sft_dataset.py.
D. Utilities & Supporting Modules
• Sharding: torchprime/sharding/shard_model.py.
• Mesh construction: torchprime/mesh/custom_mesh.py.
• Profiling & metrics: torchprime/metrics/, torchprime/utils/profiling.py.
• Lower-level tools: torchprime/tools/system_check.py, torchprime/utils/parallelism_utils.py.
E. Testing & CI
• Unit tests scattered under each pkg’s tests/.
• End-to-end tests in e2e_testing/.
• GitHub Actions workflows: lint, CPU tests, e2e tests.
F. Documentation & Examples
• Guides in docs/.
• README at repo root and per-submodule READMEs.
External Systems & Services
• Google Cloud TPU VMs / Pods.
• XPK cluster orchestration.
• Google Cloud Storage (artifact dir for outputs & checkpoints).
• BigQuery (metrics ingestion).
• HuggingFace (model checkpoints via HF_TOKEN).
• Docker registry (for custom images).
Relationships & Data Flows
• User triggers CLI → Launcher builds/pushes Docker → tp run → dispatches to TPU VMs.
• On each TPU VM:
• Hydra reads YAML configs → sets up model, dataset, parallelism.
• Data loader fetches data (HF or local).
• Trainer runs forward/backward across XLA devices (FSDP/SPMD).
• Profiling & metrics modules capture performance counters.
• Checkpoints & logs written to GCS; metrics uploaded to BigQuery.
• E2E tests simulate runs and validate logs/outputs.
Architectural Patterns & Principles
• Modular layering: CLI → Config → Core Library → TPU runtime.
• Separation of concerns: data, model definition, training loop, orchestration, utilities.
• SPMD & FSDP for scalable parallelism.
• Hydra for reproducible, override-friendly config management.