# ezelikman/star architecture

> Architecture diagram of the GitHub repository ezelikman/star, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ezelikman/star
- Repository: https://github.com/ezelikman/star

## Overview

##### Project Type and Purpose
- Type: Research/engineering library for large-scale transformer training, fine-tuning, evaluation, and serving on TPUs using JAX.
- Purpose:
– Host and orchestrate training iterations (iteration_train.py → device_train.py, create_finetune_tfrecords.py).
– Evaluate models (eval_harness.py, device_inference.py).
– Serve inference requests (device_serve.py, docker payloads/nginx).
– Provide scripts for launching Ray clusters on TPU-VMs, managing sharding (resharding_example.py), converting weights (to_hf_weights.py).

##### File Structure
- Top‐level orchestration scripts:
– train.py, iteration_train.py (driver on GCE VM).
– eval_harness.py, device_* scripts (tasks on TPUVM).
- mesh_transformer module: model construction, sharding logic, training/inference kernels.
- configs/: JSON config templates for iterative training and QA tasks.
- data pipelines: create_finetune_tfrecords.py, tfrecord_loader.py, benches and dataset folders (commonsenseqa, gsm).
- Docker assets (docker/): Dockerfile, compose files, nginx proxy, payload definitions (for serving).
- scripts/: helper shell scripts to launch Ray head/workers and RPC servers.

##### Architecture Insights from README
- Two‐tier compute:
– Driver node (GCE VM) runs orchestration, data loading, checkpoint logic.
– TPU VM workers run device_train.py and device_inference.py via Ray RPC.
- Ray-based RPC for cross-VM coordination.
- TPU-VM clusters use JAX’s xmap/pjit parallelism to shard model across 8 TPU cores (v3-8).
- Dockerized serving layer exposing HTTP endpoints via nginx + Python payloads.
- Resharding utilities for weight conversion to GPUs.

##### Main Components & Their Relationships
a. Driver Node (GCE VM)
- Runs iteration_train.py / train.py / eval_harness.py.
- Spins up Ray Head (RPC server) and Ray Workers on TPU VMs.
- Manages data paths (GCS buckets or local), checkpoint I/O.
b. TPU-VM Worker Nodes
- Ray Workers that run device_train.py (fine-tuning), device_inference.py (eval), device_serve.py (serve).
- Execute mesh_transformer.build_model, transformer_shard, sampling, train_actor.
c. Ray Cluster
- Head Node on GCE VM.
- Workers on TPU VMs.
- RPC calls from driver to workers.
d. Storage Layer
- Configs (configs/…)
- TFRecord loader → input pipeline to device_train.py.
- Checkpoints saved/loaded (mesh_transformer/checkpoint.py).
e. Dockerized Serving
- Dockerfile and compose-proxy.yaml.
- nginx proxy → forwards HTTP → Python payloads (device_serve).
- Exposes REST inference API.
f. Data Sources/Tasks
- QA datasets (commonsenseqa/, gsm/).
- gen_sums.py, arithmetic/ scripts for synthetic tasks.
g. Utility Scripts
- scripts/init_ray.sh, deploy_server.sh, create_serve_tpu.sh for automation.

##### Architectural Patterns & Design Principles
- Master-Worker (Ray) for orchestration vs. device-level compute.
- Model Parallelism (2D mesh) via JAX xmap/pjit shards.
- Separation of Concerns:
– Orchestration layer (driver).
– Data layer (TFRecords, configs).
– Compute layer (TPU shards).
– Serving layer (Docker + nginx + Python).
- Extensibility: tasks/ and benchmarks integration.

## Diagram (Mermaid)

```mermaid
flowchart TD
    client[(External Client)]:::external

    subgraph "Orchestration Layer"
        driver["Driver (GCE VM)\n(train.py, iteration_train.py,\neval_harness.py, ray_tpu.py)"]:::orchestration
        scripts["Utility Scripts\n(init_ray.sh, init_ray_v2.sh,\ninit_serve.sh, deploy_server.sh)"]:::utils
    end

    subgraph "Ray Cluster"
        head["Ray Head (GCE VM)\n(RPC Server)"]:::orchestration
        workers["Ray Workers (TPU VMs)"]:::compute
    end

    subgraph "TPU Compute Layer"
        device_train["device_train.py"]:::compute
        device_infer["device_inference.py"]:::compute
        subgraph "Mesh Transformer Module"
            builder["build_model.py"]:::compute
            shard["transformer_shard.py"]:::compute
            actor["train_actor.py"]:::compute
            sampling["sampling.py"]:::compute
            checkpoint_io["checkpoint.py"]:::storage
            tpu_utils["TPU_cluster.py"]:::compute
        end
    end

    subgraph "Storage/Data Layer"
        configs["Configs\n(base.json, qa_base.json)"]:::storage
        tf_records["TFRecord Pipeline\n(create_finetune_tfrecords.py,\ntfrecord_loader.py)"]:::storage
        datasets["Datasets\n(commonsenseqa/, gsm/)"]:::storage
        checkpoints["Checkpoint Storage (GCS)"]:::storage
    end

    subgraph "Serving Layer"
        proxy["nginx Proxy\n(compose-proxy.yaml,\nnginx_proxyvm.conf)"]:::serving
        docker_img["Docker Service\n(Dockerfile, docker-compose.yaml,\nmain.py, payloads.py, ops.py, start.sh)"]:::serving
        device_serve["device_serve.py"]:::compute
    end

    subgraph "Conversion Tools"
        reshard["resharding_example.py"]:::utils
        hf["to_hf_weights.py"]:::utils
        slim["slim_model.py"]:::utils
    end

    client -->|"HTTP"| proxy
    proxy -->|"forwards HTTP"| device_serve

    driver -->|gRPC Ray RPC| head
    scripts -->|launch cluster| driver
    head -->|schedule jobs| workers
    scripts -->|deploy serving| docker_img

    workers -->|run| device_train
    workers -->|run| device_infer
    device_train -->|calls| builder
    device_train -->|shards via| shard
    device_train -->|executes| actor
    device_infer -->|samples via| sampling
    builder -->|reads/writes| checkpoint_io
    shard -->|coordinates| tpu_utils

    device_train -->|reads| tf_records
    device_train -->|writes| checkpoints
    device_infer -->|reads| checkpoints

    driver -->|loads| configs
    driver -->|references| datasets

    docker_img -->|executes| device_serve
    docker_img -->|proxies| proxy

    reshard --> driver
    hf --> driver
    slim --> driver

    click driver "https://github.com/ezelikman/star/blob/main/train.py"
    click device_train "https://github.com/ezelikman/star/blob/main/device_train.py"
    click device_infer "https://github.com/ezelikman/star/blob/main/device_inference.py"
    click device_serve "https://github.com/ezelikman/star/blob/main/device_serve.py"
    click builder "https://github.com/ezelikman/star/blob/main/mesh_transformer/build_model.py"
    click shard "https://github.com/ezelikman/star/blob/main/mesh_transformer/transformer_shard.py"
    click actor "https://github.com/ezelikman/star/blob/main/mesh_transformer/train_actor.py"
    click sampling "https://github.com/ezelikman/star/blob/main/mesh_transformer/sampling.py"
    click checkpoint_io "https://github.com/ezelikman/star/blob/main/mesh_transformer/checkpoint.py"
    click tpu_utils "https://github.com/ezelikman/star/blob/main/mesh_transformer/TPU_cluster.py"
    click configs "https://github.com/ezelikman/star/blob/main/configs/iterative_base/base.json"
    click tf_records "https://github.com/ezelikman/star/blob/main/create_finetune_tfrecords.py"
    click datasets "https://github.com/ezelikman/star/tree/main/commonsenseqa/"
    click datasets "https://github.com/ezelikman/star/tree/main/gsm/"
    click reshard "https://github.com/ezelikman/star/blob/main/resharding_example.py"
    click hf "https://github.com/ezelikman/star/blob/main/to_hf_weights.py"
    classDef orchestration fill:#D0E8FF,stroke:#0366D6,color:#0366D6;
    classDef compute fill:#E6FFEA,stroke:#28A745,color:#22863A;
    classDef storage fill:#FFF5B1,stroke:#D4A72C,color:#735C0F;
    classDef serving fill:#FFEFE6,stroke:#DF6D0D,color:#AD2E00;
    classDef external fill:#F0F0F0,stroke:#999999,color:#333333;
    classDef utils fill:#EDEDED,stroke:#CCCCCC,color:#666666;
```
