# ai-hypercomputer/torchprime architecture

> Architecture diagram of the GitHub repository ai-hypercomputer/torchprime, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ai-hypercomputer/torchprime
- Repository: https://github.com/ai-hypercomputer/torchprime

## Overview

##### Project Type & Purpose
- torchprime is an open-source reference implementation and CLI toolset for high-performance PyTorch model training on Google Cloud TPUs. It provides:
- A CLI/launcher (`torchprime/launcher`) for local and multi-VM distributed jobs via XPK.
- A core training engine (`torchprime/torch_xla_models`) with Hydra config support, FSDP/SPMD parallelism, model definitions, and dataset integration.
- Utilities for data loading, sharding, mesh creation, profiling, metrics collection, and BigQuery reporting.
- Experimental and GPU-original model implementations for comparison.

##### Key Components & Structure
A. CLI & Orchestration
- Entry point: `torchprime/launcher/cli.py` (commands: `tp`, `torchprime`).
- Docker build & push: `launcher/Dockerfile`, `buildpush.py`.
- Remote run wrapper: `tp run` which uses XPK to broadcast to TPU VMs.
- Metrics upload: `upload_metrics_to_bq.py`.

B. Configuration Layer
- Hydra config files in `torch_xla_models/configs/` and `hf_models/configs/`.
- Overrides via CLI `key=value` or `--config-name`.
- `pyproject.toml` defines package metadata and dependencies.

C. Core Training Engine
- Training script: `torchprime/torch_xla_models/train.py`.
- Trainer abstractions: `trainer/base_trainer.py`, `trainer/sft_trainer.py`.
- Model implementations under `model/` and `model_rewriting/` for Llama, Mixtral, etc.
- Data classes: `torchprime/data/dataset.py`, `sft_dataset.py`.

D. Utilities & Supporting Modules
- Sharding: `torchprime/sharding/shard_model.py`.
- Mesh construction: `torchprime/mesh/custom_mesh.py`.
- Profiling & metrics: `torchprime/metrics/`, `torchprime/utils/profiling.py`.
- Lower-level tools: `torchprime/tools/system_check.py`, `torchprime/utils/parallelism_utils.py`.

E. Testing & CI
- Unit tests scattered under each pkg’s `tests/`.
- End-to-end tests in `e2e_testing/`.
- GitHub Actions workflows: lint, CPU tests, e2e tests.

F. Documentation & Examples
- Guides in `docs/`.
- README at repo root and per-submodule READMEs.

##### External Systems & Services
- Google Cloud TPU VMs / Pods.
- XPK cluster orchestration.
- Google Cloud Storage (artifact dir for outputs & checkpoints).
- BigQuery (metrics ingestion).
- HuggingFace (model checkpoints via HF_TOKEN).
- Docker registry (for custom images).

##### Relationships & Data Flows
- User triggers CLI → Launcher builds/pushes Docker → `tp run` → dispatches to TPU VMs.
- On each TPU VM:
- Hydra reads YAML configs → sets up model, dataset, parallelism.
- Data loader fetches data (HF or local).
- Trainer runs forward/backward across XLA devices (FSDP/SPMD).
- Profiling & metrics modules capture performance counters.
- Checkpoints & logs written to GCS; metrics uploaded to BigQuery.
- E2E tests simulate runs and validate logs/outputs.

##### Architectural Patterns & Principles
- Modular layering: CLI → Config → Core Library → TPU runtime.
- Separation of concerns: data, model definition, training loop, orchestration, utilities.
- SPMD & FSDP for scalable parallelism.
- Hydra for reproducible, override-friendly config management.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "User"
        User(["User"]):::externalUser
    end

    subgraph "CLI & Orchestration"
        CLI["CLI: tp/torchprime"]:::internal
        CLI_Build["Dockerfile & buildpush"]:::internal
        CLI_Run["run_model & tp run"]:::internal
        CLI_Metrics["upload_metrics_to_bq"]:::internal
        CLI_Benchmark["benchmark_db_util"]:::internal
    end

    subgraph "Configuration Layer"
        Config_1["torch_xla_models/configs"]:::config
        Config_2["hf_models/configs"]:::config
        Config_3["experimental/torchax_models/configs"]:::config
        PyProj["pyproject.toml"]:::config
    end

    subgraph "Core Training Engine"
        TrainScript["train.py"]:::internal
        BaseTrainer["base_trainer.py"]:::internal
        SFTTrainer["sft_trainer.py"]:::internal
        Loss["loss.py"]:::internal
        Topology["topology.py"]:::internal
        ConfigUtils["config_utils.py"]:::internal
    end

    subgraph "Model Implementations"
        BaseLM["base_causal_lm.py"]:::internal
        Llama["llama/model.py"]:::internal
        Llama4["llama4/model.py"]:::internal
        Mixtral["mixtral/model.py"]:::internal
        ModelRewriting["model_rewriting"]:::internal
    end

    subgraph "Data Modules"
        Dataset["dataset.py"]:::internal
        SFTDataset["sft_dataset.py"]:::internal
        DSConfigs["configs/dataset"]:::config
        TaskConfigs["configs/task"]:::config
    end

    subgraph "Utilities & Supporting Modules"
        Shard["shard_model.py"]:::internal
        Mesh["custom_mesh.py"]:::internal
        MetricsMod["metrics.py"]:::internal
        Profiling["profiling.py"]:::internal
        ParaUtils["parallelism_utils.py"]:::internal
        SystemCheck["system_check.py"]:::internal
    end

    subgraph "Runtime Environment"
        TPU["TPU VM/Pod"]:::runtime
    end

    subgraph "External Services"
        GCS[("GCS")]:::service
        BigQuery[("BigQuery")]:::service
        HF["HuggingFace"]:::service
        DockerReg["Docker Registry"]:::service
    end

    subgraph "Experimental & GPU"
        Exp1["torchax_models"]:::internal
        Exp2["torchrun_gpu_sft"]:::internal
        Exp3["vllm_tpu_deployment"]:::internal
    end

    subgraph "Testing & CI"
        E2E["e2e_testing"]:::internal
        UnitTests["unit tests"]:::internal
        CI1["cpu_test.yml"]:::internal
        CI2["e2e_test.yml"]:::internal
        CI3["lint.yml"]:::internal
    end

    subgraph "Documentation & Examples"
        Docs["docs"]:::internal
        README["README.md"]:::internal
        ExpReadme["torchax_models/README.md"]:::internal
        HFReadme["hf_models/README.md"]:::internal
        TXReadme["torch_xla_models/README.md"]:::internal
    end

    User -->|uses| CLI
    CLI -->|build/push| CLI_Build
    CLI_Build -->|push| DockerReg
    CLI -->|run via XPK| CLI_Run
    CLI_Run -->|dispatch| TPU
    CLI -->|upload metrics| CLI_Metrics
    CLI -->|benchmark| CLI_Benchmark

    TPU -->|reads| Config_1
    TPU -->|reads| Config_2
    TPU -->|reads| Config_3
    Config_1 -.->|dashed| TrainScript
    ConfigUtils -->|provides| TrainScript

    TPU --> TrainScript
    TrainScript --> BaseTrainer
    TrainScript --> SFTTrainer
    TrainScript --> Loss
    TrainScript --> Topology

    TrainScript --> Dataset
    Dataset -->|sharded by| Shard
    TrainScript -->|uses| Mesh
    Mesh --> TPU
    Profiling -->|profiles| TPU
    MetricsMod -->|collects| TPU

    TrainScript -->|writes| GCS
    MetricsMod -->|uploads| BigQuery
    Dataset -->|fetch from| HF

    TPU --> ParaUtils
    TPU --> SystemCheck

    E2E -->|simulate| TPU
    UnitTests -->|validate| TrainScript
    CI1 -->|runs tests| UnitTests
    CI2 -->|runs tests| E2E
    CI3 -->|runs lint| UnitTests

    classDef internal fill:#ADD8E6,stroke:#000
    classDef runtime fill:#FFA500,stroke:#000
    classDef service fill:#90EE90,stroke:#000
    classDef config fill:#DDA0DD,stroke:#000
    classDef externalUser fill:#CCCCCC,stroke:#000

    click CLI "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/launcher/cli.py"
    click CLI_Build "https://github.com/ai-hypercomputer/torchprime/tree/main/torchprime/launcher/Dockerfile"
    click CLI_Build "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/launcher/buildpush.py"
    click CLI_Metrics "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/launcher/upload_metrics_to_bq.py"
    click CLI_Benchmark "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/launcher/benchmark_db_util.py"
    click CLI_Run "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/launcher/run_model.py"
    click Config_1 "https://github.com/ai-hypercomputer/torchprime/tree/main/torchprime/torch_xla_models/configs/"
    click Config_2 "https://github.com/ai-hypercomputer/torchprime/tree/main/torchprime/hf_models/configs/"
    click Config_3 "https://github.com/ai-hypercomputer/torchprime/tree/main/torchprime/experimental/torchax_models/configs/"
    click PyProj "https://github.com/ai-hypercomputer/torchprime/blob/main/pyproject.toml"
    click TrainScript "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/torch_xla_models/train.py"
    click BaseTrainer "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/torch_xla_models/trainer/base_trainer.py"
    click SFTTrainer "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/torch_xla_models/trainer/sft_trainer.py"
    click Loss "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/torch_xla_models/loss.py"
    click Topology "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/torch_xla_models/topology.py"
    click ConfigUtils "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/torch_xla_models/utils/config_utils.py"
    click BaseLM "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/torch_xla_models/model/base_causal_lm.py"
    click Llama "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/torch_xla_models/model/llama/model.py"
    click Llama4 "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/torch_xla_models/model/llama4/model.py"
    click Mixtral "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/torch_xla_models/model/mixtral/model.py"
    click ModelRewriting "https://github.com/ai-hypercomputer/torchprime/tree/main/torchprime/torch_xla_models/model_rewriting/"
    click Dataset "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/data/dataset.py"
    click SFTDataset "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/data/sft_dataset.py"
    click DSConfigs "https://github.com/ai-hypercomputer/torchprime/tree/main/torchprime/torch_xla_models/configs/dataset/"
    click TaskConfigs "https://github.com/ai-hypercomputer/torchprime/tree/main/torchprime/torch_xla_models/configs/task/"
    click Shard "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/sharding/shard_model.py"
    click Mesh "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/mesh/custom_mesh.py"
    click MetricsMod "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/metrics/metrics.py"
    click Profiling "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/utils/profiling.py"
    click ParaUtils "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/utils/parallelism_utils.py"
    click SystemCheck "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/tools/system_check.py"
    click Exp1 "https://github.com/ai-hypercomputer/torchprime/tree/main/torchprime/experimental/torchax_models/"
    click Exp2 "https://github.com/ai-hypercomputer/torchprime/tree/main/torchprime/experimental/torchrun_gpu_sft/"
    click Exp3 "https://github.com/ai-hypercomputer/torchprime/tree/main/torchprime/experimental/vllm_tpu_deployment/"
    click E2E "https://github.com/ai-hypercomputer/torchprime/tree/main/e2e_testing/"
    click UnitTests "https://github.com/ai-hypercomputer/torchprime/tree/main/torchprime/**/tests/"
    click CI1 "https://github.com/ai-hypercomputer/torchprime/blob/main/.github/workflows/cpu_test.yml"
    click CI2 "https://github.com/ai-hypercomputer/torchprime/blob/main/.github/workflows/e2e_test.yml"
    click CI3 "https://github.com/ai-hypercomputer/torchprime/blob/main/.github/workflows/lint.yml"
    click Docs "https://github.com/ai-hypercomputer/torchprime/tree/main/docs/"
    click README "https://github.com/ai-hypercomputer/torchprime/blob/main/README.md"
    click ExpReadme "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/experimental/torchax_models/README.md"
    click HFReadme "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/hf_models/README.md"
    click TXReadme "https://github.com/ai-hypercomputer/torchprime/blob/main/torchprime/torch_xla_models/README.md"
```
