How ezelikman/star works
Project Type and Purpose
• Type: Research/engineering library for large-scale transformer training, fine-tuning, evaluation, and serving on TPUs using JAX.
• Purpose:
– Host and orchestrate training iterations (iteration_train.py → device_train.py, create_finetune_tfrecords.py).
– Evaluate models (eval_harness.py, device_inference.py).
– Serve inference requests (device_serve.py, docker payloads/nginx).
– Provide scripts for launching Ray clusters on TPU-VMs, managing sharding (resharding_example.py), converting weights (to_hf_weights.py).
File Structure
• Top‐level orchestration scripts:
– train.py, iteration_train.py (driver on GCE VM).
– eval_harness.py, device_* scripts (tasks on TPUVM).
• mesh_transformer module: model construction, sharding logic, training/inference kernels.
• configs/: JSON config templates for iterative training and QA tasks.
• data pipelines: create_finetune_tfrecords.py, tfrecord_loader.py, benches and dataset folders (commonsenseqa, gsm).
• Docker assets (docker/): Dockerfile, compose files, nginx proxy, payload definitions (for serving).
• scripts/: helper shell scripts to launch Ray head/workers and RPC servers.
Architecture Insights from README
• Two‐tier compute:
– Driver node (GCE VM) runs orchestration, data loading, checkpoint logic.
– TPU VM workers run device_train.py and device_inference.py via Ray RPC.
• Ray-based RPC for cross-VM coordination.
• TPU-VM clusters use JAX’s xmap/pjit parallelism to shard model across 8 TPU cores (v3-8).
• Dockerized serving layer exposing HTTP endpoints via nginx + Python payloads.
• Resharding utilities for weight conversion to GPUs.
Main Components & Their Relationships
a. Driver Node (GCE VM)
• Runs iteration_train.py / train.py / eval_harness.py.
• Spins up Ray Head (RPC server) and Ray Workers on TPU VMs.
• Manages data paths (GCS buckets or local), checkpoint I/O.
b. TPU-VM Worker Nodes
• Ray Workers that run device_train.py (fine-tuning), device_inference.py (eval), device_serve.py (serve).
• Execute mesh_transformer.build_model, transformer_shard, sampling, train_actor.
c. Ray Cluster
• Head Node on GCE VM.
• Workers on TPU VMs.
• RPC calls from driver to workers.
d. Storage Layer
• Configs (configs/…)
• TFRecord loader → input pipeline to device_train.py.
• Checkpoints saved/loaded (mesh_transformer/checkpoint.py).
e. Dockerized Serving
• Dockerfile and compose-proxy.yaml.
• nginx proxy → forwards HTTP → Python payloads (device_serve).
• Exposes REST inference API.
f. Data Sources/Tasks
• QA datasets (commonsenseqa/, gsm/).
• gen_sums.py, arithmetic/ scripts for synthetic tasks.
g. Utility Scripts
• scripts/init_ray.sh, deploy_server.sh, create_serve_tpu.sh for automation.
Architectural Patterns & Design Principles
• Master-Worker (Ray) for orchestration vs. device-level compute.
• Model Parallelism (2D mesh) via JAX xmap/pjit shards.
• Separation of Concerns:
– Orchestration layer (driver).
– Data layer (TFRecords, configs).
– Compute layer (TPU shards).
– Serving layer (Docker + nginx + Python).
• Extensibility: tasks/ and benchmarks integration.