# meta-pytorch/torchforge architecture

> Architecture diagram of the GitHub repository meta-pytorch/torchforge, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/meta-pytorch/torchforge
- Repository: https://github.com/meta-pytorch/torchforge

## Overview

torchforge is a PyTorch-native agentic reinforcement-learning library that separates algorithm work from distributed execution infrastructure. The principal workflow is the GRPO application: configure and provision resources, sample prompts, generate responses, grade rewards, compute advantages, and train from rollouts. The repository also includes an SFT training entry point, reusable dataset/tokenization and RL-loss components, scalable actor services, and metrics. The GRPO excerpt supports the main orchestration links; several runtime modules are represented from the source index and README capabilities, so unverified links are omitted.

Main entry points: [`apps/grpo/main.py`](https://github.com/meta-pytorch/torchforge/blob/main/apps/grpo/main.py) (GRPO runner), [`apps/sft/main.py`](https://github.com/meta-pytorch/torchforge/blob/main/apps/sft/main.py) (SFT runner), [`src/forge/actors/vllm/v1/generator.py`](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/actors/vllm/v1/generator.py) (vLLM generation), [`src/forge/controller/service/service.py`](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/controller/service/service.py) (Actor service)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflow["Training workflows"]
  node_grpo_app["GRPO runner<br/>[main.py]"]
  node_sft_app["SFT runner<br/>[main.py]"]
end

subgraph group_rl["RL algorithms"]
  node_grader["Reward grading<br/>[grading.py]"]
  node_advantage["Advantage calculation<br/>[advantage.py]"]
  node_rl_collate["RL collation<br/>[collate.py]"]
  node_losses["RL loss variants"]
  node_grading["RL grading<br/>[grading.py]"]
end

subgraph group_data["Data and observability"]
  node_grpo_data["Rollout dataset<br/>[data.py]"]
  node_prompt_types["Prompt models<br/>[prompt.py]"]
  node_completion_types["Completion models<br/>[completion.py]"]
  node_datasets["Dataset loaders"]
  node_tokenizer["Tokenizer<br/>[tokenizer.py]"]
  node_collate["SFT collation<br/>[collate.py]"]
  node_metrics["Metric aggregation<br/>[metrics.py]"]
  node_metric_actors["Metric actors<br/>[metric_actors.py]"]
  node_perf["Performance tracking<br/>[perf_tracker.py]"]
end

subgraph group_runtime["Distributed runtime"]
  node_provisioner["Resource provisioner<br/>[provisioner.py]"]
  node_actor["Forge actor<br/>[actor.py]"]
  node_service["Actor service<br/>[service.py]"]
  node_router["Request router<br/>[router.py]"]
  node_replica["Service replica<br/>[replica.py]"]
  node_env["Runtime settings<br/>[env.py]"]
end

subgraph group_inference["Generation and training"]
  node_generator["Generator actor<br/>[generator.py]"]
  node_vllm_generator["vLLM generation<br/>[generator.py]"]
  node_reference["Reference model<br/>[reference_model.py]"]
  node_replay[("Replay buffer<br/>[replay_buffer.py]")]
  node_trainer["Titan trainer<br/>[titan.py]"]
  node_trainer_api["Trainer API<br/>[trainer.py]"]
end

node_engineer(("Researcher"))

node_engineer -->|"runs"| node_grpo_app
node_engineer -->|"runs"| node_sft_app
node_grpo_app -->|"provisions"| node_provisioner
node_grpo_app -->|"samples"| node_grpo_data
node_grpo_app -->|"grades responses"| node_grader
node_grpo_app -->|"computes advantages"| node_advantage
node_grpo_app -->|"launches rollouts"| node_generator
node_grpo_app -->|"uses reference"| node_reference
node_grpo_app -->|"uses buffer"| node_replay
node_grpo_app -->|"trains model"| node_trainer
node_grpo_app -->|"uses completions"| node_completion_types
node_grpo_app -->|"initializes logging"| node_metric_actors
node_grpo_app -->|"tracks performance"| node_perf
node_sft_app -->|"loads datasets"| node_datasets
node_sft_app -->|"tokenizes data"| node_tokenizer
node_sft_app -->|"collates batches"| node_collate
node_sft_app -->|"uses runtime"| node_provisioner
node_grpo_data -->|"creates prompts"| node_prompt_types
node_generator -->|"delegates generation"| node_vllm_generator
node_vllm_generator -->|"accepts prompts"| node_prompt_types
node_vllm_generator -->|"produces completions"| node_completion_types
node_trainer -->|"uses training API"| node_trainer_api
node_trainer -->|"applies loss"| node_losses
node_provisioner -->|"deploys actors"| node_actor
node_actor -->|"creates service"| node_service
node_service -->|"routes requests"| node_router
node_service -->|"enqueues requests"| node_replica
node_router -->|"selects replica"| node_replica
node_metric_actors -->|"aggregates metrics"| node_metrics
node_perf -->|"records metrics"| node_metrics
node_vllm_generator -->|"reads settings"| node_env

click node_grpo_app "https://github.com/meta-pytorch/torchforge/blob/main/apps/grpo/main.py"
click node_sft_app "https://github.com/meta-pytorch/torchforge/blob/main/apps/sft/main.py"
click node_grpo_data "https://github.com/meta-pytorch/torchforge/blob/main/apps/grpo/data.py"
click node_grader "https://github.com/meta-pytorch/torchforge/blob/main/apps/grpo/grading.py"
click node_prompt_types "https://github.com/meta-pytorch/torchforge/blob/main/src/forge/data_models/prompt.py"
click node_completion_types "https://github.com/meta-pytorch/torchforge/blob/main/src/forge/data_models/completion.py"
click node_advantage "https://github.com/meta-pytorch/torchforge/blob/main/src/forge/rl/advantage.py"
click node_rl_collate "https://github.com/meta-pytorch/torchforge/blob/main/src/forge/rl/collate.py"
click node_losses "https://github.com/meta-pytorch/torchforge/tree/main/src/forge/rl/loss"
click node_grading "https://github.com/meta-pytorch/torchforge/blob/main/src/forge/rl/grading.py"
click node_datasets "https://github.com/meta-pytorch/torchforge/tree/main/src/forge/data/datasets"
click node_tokenizer "https://github.com/meta-pytorch/torchforge/blob/main/src/forge/data/tokenizer.py"
click node_collate "https://github.com/meta-pytorch/torchforge/blob/main/src/forge/data/collate.py"
click node_generator "https://github.com/meta-pytorch/torchforge/blob/main/src/forge/actors/generator.py"
click node_vllm_generator "https://github.com/meta-pytorch/torchforge/blob/main/src/forge/actors/vllm/v1/generator.py"
click node_reference "https://github.com/meta-pytorch/torchforge/blob/main/src/forge/actors/reference_model.py"
click node_replay "https://github.com/meta-pytorch/torchforge/blob/main/src/forge/actors/replay_buffer.py"
click node_trainer "https://github.com/meta-pytorch/torchforge/blob/main/src/forge/actors/trainer/titan.py"
click node_trainer_api "https://github.com/meta-pytorch/torchforge/blob/main/src/forge/api/trainer.py"
click node_provisioner "https://github.com/meta-pytorch/torchforge/blob/main/src/forge/controller/provisioner.py"
click node_actor "https://github.com/meta-pytorch/torchforge/blob/main/src/forge/controller/actor.py"
click node_service "https://github.com/meta-pytorch/torchforge/blob/main/src/forge/controller/service/service.py"
click node_router "https://github.com/meta-pytorch/torchforge/blob/main/src/forge/controller/service/router.py"
click node_replica "https://github.com/meta-pytorch/torchforge/blob/main/src/forge/controller/service/replica.py"
click node_metrics "https://github.com/meta-pytorch/torchforge/blob/main/src/forge/observability/metrics.py"
click node_metric_actors "https://github.com/meta-pytorch/torchforge/blob/main/src/forge/observability/metric_actors.py"
click node_perf "https://github.com/meta-pytorch/torchforge/blob/main/src/forge/observability/perf_tracker.py"
click node_env "https://github.com/meta-pytorch/torchforge/blob/main/src/forge/env.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_grpo_app,node_sft_app toneBlue
class node_grader,node_advantage,node_rl_collate,node_losses,node_grading toneAmber
class node_grpo_data,node_prompt_types,node_completion_types,node_datasets,node_tokenizer,node_collate,node_metrics,node_metric_actors,node_perf toneMint
class node_provisioner,node_actor,node_service,node_router,node_replica,node_env toneRose
class node_generator,node_vllm_generator,node_reference,node_replay,node_trainer,node_trainer_api,node_engineer toneIndigo
```

## Components

### Training workflows

- [GRPO runner](https://github.com/meta-pytorch/torchforge/blob/main/apps/grpo/main.py): `apps/grpo/main.py`
- [SFT runner](https://github.com/meta-pytorch/torchforge/blob/main/apps/sft/main.py): `apps/sft/main.py`

### RL algorithms

- [Reward grading](https://github.com/meta-pytorch/torchforge/blob/main/apps/grpo/grading.py): `apps/grpo/grading.py`
- [Advantage calculation](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/rl/advantage.py): `src/forge/rl/advantage.py`
- [RL collation](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/rl/collate.py): `src/forge/rl/collate.py`
- [RL loss variants](https://github.com/meta-pytorch/torchforge/tree/main/src/forge/rl/loss): `src/forge/rl/loss`
- [RL grading](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/rl/grading.py): `src/forge/rl/grading.py`

### Data and observability

- [Rollout dataset](https://github.com/meta-pytorch/torchforge/blob/main/apps/grpo/data.py): `apps/grpo/data.py`
- [Prompt models](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/data_models/prompt.py): `src/forge/data_models/prompt.py`
- [Completion models](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/data_models/completion.py): `src/forge/data_models/completion.py`
- [Dataset loaders](https://github.com/meta-pytorch/torchforge/tree/main/src/forge/data/datasets): `src/forge/data/datasets`
- [Tokenizer](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/data/tokenizer.py): `src/forge/data/tokenizer.py`
- [SFT collation](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/data/collate.py): `src/forge/data/collate.py`
- [Metric aggregation](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/observability/metrics.py): `src/forge/observability/metrics.py`
- [Metric actors](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/observability/metric_actors.py): `src/forge/observability/metric_actors.py`
- [Performance tracking](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/observability/perf_tracker.py): `src/forge/observability/perf_tracker.py`

### Distributed runtime

- [Resource provisioner](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/controller/provisioner.py): `src/forge/controller/provisioner.py`
- [Forge actor](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/controller/actor.py): `src/forge/controller/actor.py`
- [Actor service](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/controller/service/service.py): `src/forge/controller/service/service.py`
- [Request router](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/controller/service/router.py): `src/forge/controller/service/router.py`
- [Service replica](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/controller/service/replica.py): `src/forge/controller/service/replica.py`
- [Runtime settings](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/env.py): `src/forge/env.py`

### Generation and training

- [Generator actor](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/actors/generator.py): `src/forge/actors/generator.py`
- [vLLM generation](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/actors/vllm/v1/generator.py): `src/forge/actors/vllm/v1/generator.py`
- [Reference model](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/actors/reference_model.py): `src/forge/actors/reference_model.py`
- [Replay buffer](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/actors/replay_buffer.py): `src/forge/actors/replay_buffer.py`
- [Titan trainer](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/actors/trainer/titan.py): `src/forge/actors/trainer/titan.py`
- [Trainer API](https://github.com/meta-pytorch/torchforge/blob/main/src/forge/api/trainer.py): `src/forge/api/trainer.py`

### Other components

- **Researcher**

## Connections

- Researcher → GRPO runner: runs (evidence: `README.md`)
- Researcher → SFT runner: runs (evidence: `README.md`)
- GRPO runner → Resource provisioner: provisions (evidence: `apps/grpo/main.py`)
- GRPO runner → Rollout dataset: samples (evidence: `apps/grpo/main.py`)
- GRPO runner → Reward grading: grades responses (evidence: `apps/grpo/main.py`)
- GRPO runner → Advantage calculation: computes advantages (evidence: `apps/grpo/main.py`)
- GRPO runner → Generator actor: launches rollouts (evidence: `apps/grpo/main.py`)
- GRPO runner → Reference model: uses reference (evidence: `apps/grpo/main.py`)
- GRPO runner → Replay buffer: uses buffer (evidence: `apps/grpo/main.py`)
- GRPO runner → Titan trainer: trains model (evidence: `apps/grpo/main.py`)
- GRPO runner → Completion models: uses completions (evidence: `apps/grpo/main.py`)
- GRPO runner → Metric actors: initializes logging (evidence: `apps/grpo/main.py`)
- GRPO runner → Performance tracking: tracks performance (evidence: `apps/grpo/main.py`)
- SFT runner → Dataset loaders: loads datasets (evidence: `apps/sft/main.py`)
- SFT runner → Tokenizer: tokenizes data (evidence: `apps/sft/main.py`)
- SFT runner → SFT collation: collates batches (evidence: `apps/sft/main.py`)
- SFT runner → Resource provisioner: uses runtime (evidence: `apps/sft/main.py`)
- Rollout dataset → Prompt models: creates prompts (evidence: `apps/grpo/data.py`)
- Generator actor → vLLM generation: delegates generation
- vLLM generation → Prompt models: accepts prompts (evidence: `src/forge/actors/vllm/v1/generator.py`)
- vLLM generation → Completion models: produces completions (evidence: `src/forge/actors/vllm/v1/generator.py`)
- Titan trainer → Trainer API: uses training API (evidence: `src/forge/actors/trainer/titan.py`)
- Titan trainer → RL loss variants: applies loss (evidence: `src/forge/actors/trainer/titan.py`)
- Resource provisioner → Forge actor: deploys actors (evidence: `src/forge/controller/provisioner.py`)
- Forge actor → Actor service: creates service (evidence: `src/forge/controller/actor.py`)
- Actor service → Request router: routes requests (evidence: `src/forge/controller/service/service.py`)
- Actor service → Service replica: enqueues requests (evidence: `src/forge/controller/service/service.py`)
- Request router → Service replica: selects replica (evidence: `src/forge/controller/service/router.py`)
- Metric actors → Metric aggregation: aggregates metrics (evidence: `src/forge/observability/metric_actors.py`)
- Performance tracking → Metric aggregation: records metrics (evidence: `src/forge/observability/perf_tracker.py`)
- vLLM generation → Runtime settings: reads settings (evidence: `src/forge/actors/vllm/v1/generator.py`)
