# tencent-hunyuan/unirl architecture

> Architecture diagram of the GitHub repository tencent-hunyuan/unirl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tencent-hunyuan/unirl
- Repository: https://github.com/tencent-hunyuan/unirl

## Overview

UniRL is a framework for reinforcement-learning post-training of multimodal models. The documented loop takes supervised or prompt data, generates rollouts, scores them, computes advantages and updates the policy, then synchronizes weights to rollout workers. Its model and rollout support spans autoregressive, diffusion, video, and composed workflows; distributed execution and reward services are shared runtime layers. Broad workflow links below follow the README; sampled source confirms several core contracts, but unsampled trainer and algorithm internals are not inferred in detail.

Main entry points: [`unirl/trainer`](https://github.com/tencent-hunyuan/unirl/tree/main/unirl/trainer) (Trainers), [`unirl/rollout/engine`](https://github.com/tencent-hunyuan/unirl/tree/main/unirl/rollout/engine) (Rollout engines), [`unirl/models/types/pipeline.py`](https://github.com/tencent-hunyuan/unirl/blob/main/unirl/models/types/pipeline.py) (Model pipelines), [`unirl/distributed/group/dispatch.py`](https://github.com/tencent-hunyuan/unirl/blob/main/unirl/distributed/group/dispatch.py) (Worker dispatch)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_training["Training"]
  node_data["Training data"]
  node_trainers["Trainers"]
  node_algorithms["RL algorithms"]
  node_sft["SFT training<br/>[track_builder.py]"]
end

subgraph group_rollout["Rollout"]
  node_rollout_engine["Rollout engines"]
  node_agentic["Agentic rollout<br/>[engine.py]"]
  node_composed["Composed rollout<br/>[engine.py]"]
end

subgraph group_models["Models"]
  node_model_pipeline["Model pipelines<br/>[pipeline.py]"]
  node_wan_bundle["WAN model bundle<br/>[bundle.py]"]
  node_bagel_pipeline["BAGEL pipeline<br/>[pipeline.py]"]
end

subgraph group_runtime["Shared Runtime"]
  node_sample["Sample format<br/>[sample.py]"]
  node_sampling["Sampling parameters<br/>[sampling.py]"]
  node_sde["SDE kernels<br/>[kernels.py]"]
  node_dispatch["Worker dispatch<br/>[dispatch.py]"]
  node_remote["Remote workers<br/>[remote.py]"]
  node_tensor_batch["Tensor batches<br/>[batch.py]"]
  node_weight_sync["Weight synchronization"]
end

subgraph group_rewards["Reward Scoring"]
  node_reward["Reward functions"]
  node_reward_service["Reward service<br/>[server.py]"]
end

node_user(("Training user"))

node_user -->|"configures"| node_trainers
node_data -->|"provides data"| node_trainers
node_trainers -->|"requests samples"| node_rollout_engine
node_trainers -->|"scores samples"| node_reward
node_trainers -->|"computes updates"| node_algorithms
node_trainers -->|"syncs weights"| node_weight_sync
node_trainers -->|"runs SFT"| node_sft
node_rollout_engine -->|"generates with"| node_model_pipeline
node_rollout_engine -->|"fills request"| node_sample
node_model_pipeline -->|"consumes"| node_sample
node_bagel_pipeline -.->|"model example"| node_wan_bundle
node_rollout_engine -->|"dispatches work"| node_dispatch
node_dispatch -->|"uses handles"| node_remote
node_dispatch -->|"chunks batches"| node_tensor_batch
node_agentic -->|"implements engine"| node_rollout_engine
node_composed -->|"implements engine"| node_rollout_engine
node_model_pipeline -.->|"uses transitions"| node_sde
node_sampling -->|"configures generation"| node_sample
node_reward -.->|"can use service"| node_reward_service

click node_data "https://github.com/tencent-hunyuan/unirl/tree/main/unirl/data"
click node_trainers "https://github.com/tencent-hunyuan/unirl/tree/main/unirl/trainer"
click node_algorithms "https://github.com/tencent-hunyuan/unirl/tree/main/unirl/algorithms"
click node_sft "https://github.com/tencent-hunyuan/unirl/blob/main/unirl/train/sft/track_builder.py"
click node_rollout_engine "https://github.com/tencent-hunyuan/unirl/tree/main/unirl/rollout/engine"
click node_agentic "https://github.com/tencent-hunyuan/unirl/blob/main/unirl/rollout/engine/agentic/engine.py"
click node_composed "https://github.com/tencent-hunyuan/unirl/blob/main/unirl/rollout/engine/composed/engine.py"
click node_sample "https://github.com/tencent-hunyuan/unirl/blob/main/unirl/types/sample.py"
click node_sampling "https://github.com/tencent-hunyuan/unirl/blob/main/unirl/types/sampling.py"
click node_model_pipeline "https://github.com/tencent-hunyuan/unirl/blob/main/unirl/models/types/pipeline.py"
click node_wan_bundle "https://github.com/tencent-hunyuan/unirl/blob/main/unirl/models/wan21/bundle.py"
click node_bagel_pipeline "https://github.com/tencent-hunyuan/unirl/blob/main/unirl/models/bagel/pipeline.py"
click node_sde "https://github.com/tencent-hunyuan/unirl/blob/main/unirl/sde/kernels.py"
click node_dispatch "https://github.com/tencent-hunyuan/unirl/blob/main/unirl/distributed/group/dispatch.py"
click node_remote "https://github.com/tencent-hunyuan/unirl/blob/main/unirl/distributed/group/remote.py"
click node_tensor_batch "https://github.com/tencent-hunyuan/unirl/blob/main/unirl/distributed/tensor/batch.py"
click node_weight_sync "https://github.com/tencent-hunyuan/unirl/tree/main/unirl/distributed/weight_sync"
click node_reward "https://github.com/tencent-hunyuan/unirl/tree/main/unirl/reward"
click node_reward_service "https://github.com/tencent-hunyuan/unirl/blob/main/unirl-reward-service/reward_service/server.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_data,node_trainers,node_algorithms,node_sft,node_user toneBlue
class node_rollout_engine,node_agentic,node_composed toneAmber
class node_model_pipeline,node_wan_bundle,node_bagel_pipeline toneMint
class node_sample,node_sampling,node_sde,node_dispatch,node_remote,node_tensor_batch,node_weight_sync toneRose
class node_reward,node_reward_service toneIndigo
```

## Components

### Training

- [Training data](https://github.com/tencent-hunyuan/unirl/tree/main/unirl/data): `unirl/data`
- [Trainers](https://github.com/tencent-hunyuan/unirl/tree/main/unirl/trainer): `unirl/trainer`
- [RL algorithms](https://github.com/tencent-hunyuan/unirl/tree/main/unirl/algorithms): `unirl/algorithms`
- [SFT training](https://github.com/tencent-hunyuan/unirl/blob/main/unirl/train/sft/track_builder.py): `unirl/train/sft/track_builder.py`

### Rollout

- [Rollout engines](https://github.com/tencent-hunyuan/unirl/tree/main/unirl/rollout/engine): `unirl/rollout/engine`
- [Agentic rollout](https://github.com/tencent-hunyuan/unirl/blob/main/unirl/rollout/engine/agentic/engine.py): `unirl/rollout/engine/agentic/engine.py`
- [Composed rollout](https://github.com/tencent-hunyuan/unirl/blob/main/unirl/rollout/engine/composed/engine.py): `unirl/rollout/engine/composed/engine.py`

### Models

- [Model pipelines](https://github.com/tencent-hunyuan/unirl/blob/main/unirl/models/types/pipeline.py): `unirl/models/types/pipeline.py`
- [WAN model bundle](https://github.com/tencent-hunyuan/unirl/blob/main/unirl/models/wan21/bundle.py): `unirl/models/wan21/bundle.py`
- [BAGEL pipeline](https://github.com/tencent-hunyuan/unirl/blob/main/unirl/models/bagel/pipeline.py): `unirl/models/bagel/pipeline.py`

### Shared Runtime

- [Sample format](https://github.com/tencent-hunyuan/unirl/blob/main/unirl/types/sample.py): `unirl/types/sample.py`
- [Sampling parameters](https://github.com/tencent-hunyuan/unirl/blob/main/unirl/types/sampling.py): `unirl/types/sampling.py`
- [SDE kernels](https://github.com/tencent-hunyuan/unirl/blob/main/unirl/sde/kernels.py): `unirl/sde/kernels.py`
- [Worker dispatch](https://github.com/tencent-hunyuan/unirl/blob/main/unirl/distributed/group/dispatch.py): `unirl/distributed/group/dispatch.py`
- [Remote workers](https://github.com/tencent-hunyuan/unirl/blob/main/unirl/distributed/group/remote.py): `unirl/distributed/group/remote.py`
- [Tensor batches](https://github.com/tencent-hunyuan/unirl/blob/main/unirl/distributed/tensor/batch.py): `unirl/distributed/tensor/batch.py`
- [Weight synchronization](https://github.com/tencent-hunyuan/unirl/tree/main/unirl/distributed/weight_sync): `unirl/distributed/weight_sync`

### Reward Scoring

- [Reward functions](https://github.com/tencent-hunyuan/unirl/tree/main/unirl/reward): `unirl/reward`
- [Reward service](https://github.com/tencent-hunyuan/unirl/blob/main/unirl-reward-service/reward_service/server.py): `unirl-reward-service/reward_service/server.py`

### Other components

- **Training user**

## Connections

- Training user → Trainers: configures (evidence: `README.md`)
- Training data → Trainers: provides data (evidence: `README.md`)
- Trainers → Rollout engines: requests samples (evidence: `README.md`)
- Trainers → Reward functions: scores samples (evidence: `README.md`)
- Trainers → RL algorithms: computes updates (evidence: `README.md`)
- Trainers → Weight synchronization: syncs weights (evidence: `README.md`)
- Trainers → SFT training: runs SFT (evidence: `README.md`)
- Rollout engines → Model pipelines: generates with (evidence: `README.md`)
- Rollout engines → Sample format: fills request (evidence: `unirl/rollout/engine/base.py`)
- Model pipelines → Sample format: consumes (evidence: `unirl/models/types/pipeline.py`)
- BAGEL pipeline → WAN model bundle: model example
- Rollout engines → Worker dispatch: dispatches work (evidence: `unirl/rollout/engine/base.py`)
- Worker dispatch → Remote workers: uses handles (evidence: `unirl/distributed/group/dispatch.py`)
- Worker dispatch → Tensor batches: chunks batches (evidence: `unirl/distributed/group/dispatch.py`)
- Agentic rollout → Rollout engines: implements engine (evidence: `unirl/rollout/engine/agentic/engine.py`)
- Composed rollout → Rollout engines: implements engine (evidence: `unirl/rollout/engine/composed/engine.py`)
- Model pipelines → SDE kernels: uses transitions
- Sampling parameters → Sample format: configures generation (evidence: `unirl/types/sample.py`)
- Reward functions → Reward service: can use service (evidence: `README.md`)
