How meta-pytorch/torchforge works
torchforge is a PyTorch-native agentic reinforcement-learning library that separates algorithm work from distributed execution infrastructure. The principal workflow is the GRPO application: configure and provision resources, sample prompts, generate responses, grade rewards, compute advantages, and train from rollouts. The repository also includes an SFT training entry point, reusable dataset/tokenization and RL-loss components, scalable actor services, and metrics. The GRPO excerpt supports the main orchestration links; several runtime modules are represented from the source index and README capabilities, so unverified links are omitted.
Main entry points: grpo/main.py GRPO runner · sft/main.py SFT runner · generator.py vLLM generation · service.py Actor service
Components 29
Training workflows
- GRPO runner
apps/grpo/main.py - SFT runner
apps/sft/main.py
RL algorithms
- Reward grading
apps/grpo/grading.py - Advantage calculation
src/forge/rl/advantage.py - RL collation
src/forge/rl/collate.py - RL loss variants
src/forge/rl/loss - RL grading
src/forge/rl/grading.py
Data and observability
- Rollout dataset
apps/grpo/data.py - Prompt models
src/forge/data_models/prompt.py - Completion models
src/forge/data_models/completion.py - Dataset loaders
src/forge/data/datasets - Tokenizer
src/forge/data/tokenizer.py - SFT collation
src/forge/data/collate.py - Metric aggregation
src/forge/observability/metrics.py - Metric actors
src/forge/observability/metric_actors.py - Performance tracking
src/forge/observability/perf_tracker.py
Distributed runtime
- Resource provisioner
src/forge/controller/provisioner.py - Forge actor
src/forge/controller/actor.py - Actor service
src/forge/controller/service/service.py - Request router
src/forge/controller/service/router.py - Service replica
src/forge/controller/service/replica.py - Runtime settings
src/forge/env.py
Generation and training
- Generator actor
src/forge/actors/generator.py - vLLM generation
src/forge/actors/vllm/v1/generator.py - Reference model
src/forge/actors/reference_model.py - Replay buffer
src/forge/actors/replay_buffer.py - Titan trainer
src/forge/actors/trainer/titan.py - Trainer API
src/forge/api/trainer.py
Other components
- Researcher
Connections 31
- Researcher → GRPO runner: runs
README.md - Researcher → SFT runner: runs
README.md - GRPO runner → Resource provisioner: provisions
apps/grpo/main.py - GRPO runner → Rollout dataset: samples
apps/grpo/main.py - GRPO runner → Reward grading: grades responses
apps/grpo/main.py - GRPO runner → Advantage calculation: computes advantages
apps/grpo/main.py - GRPO runner → Generator actor: launches rollouts
apps/grpo/main.py - GRPO runner → Reference model: uses reference
apps/grpo/main.py - GRPO runner → Replay buffer: uses buffer
apps/grpo/main.py - GRPO runner → Titan trainer: trains model
apps/grpo/main.py - GRPO runner → Completion models: uses completions
apps/grpo/main.py - GRPO runner → Metric actors: initializes logging
apps/grpo/main.py - GRPO runner → Performance tracking: tracks performance
apps/grpo/main.py - SFT runner → Dataset loaders: loads datasets
apps/sft/main.py - SFT runner → Tokenizer: tokenizes data
apps/sft/main.py - SFT runner → SFT collation: collates batches
apps/sft/main.py - SFT runner → Resource provisioner: uses runtime
apps/sft/main.py - Rollout dataset → Prompt models: creates prompts
apps/grpo/data.py - Generator actor → vLLM generation: delegates generation
- vLLM generation → Prompt models: accepts prompts
src/forge/actors/vllm/v1/generator.py - vLLM generation → Completion models: produces completions
src/forge/actors/vllm/v1/generator.py - Titan trainer → Trainer API: uses training API
src/forge/actors/trainer/titan.py - Titan trainer → RL loss variants: applies loss
src/forge/actors/trainer/titan.py - Resource provisioner → Forge actor: deploys actors
src/forge/controller/provisioner.py - Forge actor → Actor service: creates service
src/forge/controller/actor.py - Actor service → Request router: routes requests
src/forge/controller/service/service.py - Actor service → Service replica: enqueues requests
src/forge/controller/service/service.py - Request router → Service replica: selects replica
src/forge/controller/service/router.py - Metric actors → Metric aggregation: aggregates metrics
src/forge/observability/metric_actors.py - Performance tracking → Metric aggregation: records metrics
src/forge/observability/perf_tracker.py - vLLM generation → Runtime settings: reads settings
src/forge/actors/vllm/v1/generator.py