How tencent-hunyuan/unirl works
UniRL is a framework for reinforcement-learning post-training of multimodal models. The documented loop takes supervised or prompt data, generates rollouts, scores them, computes advantages and updates the policy, then synchronizes weights to rollout workers. Its model and rollout support spans autoregressive, diffusion, video, and composed workflows; distributed execution and reward services are shared runtime layers. Broad workflow links below follow the README; sampled source confirms several core contracts, but unsampled trainer and algorithm internals are not inferred in detail.
Main entry points: trainer Trainers · engine Rollout engines · pipeline.py Model pipelines · dispatch.py Worker dispatch
Components 20
Training
- Training data
unirl/data - Trainers
unirl/trainer - RL algorithms
unirl/algorithms - SFT training
unirl/train/sft/track_builder.py
Rollout
- Rollout engines
unirl/rollout/engine - Agentic rollout
unirl/rollout/engine/agentic/engine.py - Composed rollout
unirl/rollout/engine/composed/engine.py
Models
- Model pipelines
unirl/models/types/pipeline.py - WAN model bundle
unirl/models/wan21/bundle.py - BAGEL pipeline
unirl/models/bagel/pipeline.py
Shared Runtime
- Sample format
unirl/types/sample.py - Sampling parameters
unirl/types/sampling.py - SDE kernels
unirl/sde/kernels.py - Worker dispatch
unirl/distributed/group/dispatch.py - Remote workers
unirl/distributed/group/remote.py - Tensor batches
unirl/distributed/tensor/batch.py - Weight synchronization
unirl/distributed/weight_sync
Reward Scoring
- Reward functions
unirl/reward - Reward service
unirl-reward-service/reward_service/server.py
Other components
- Training user
Connections 19
- Training user → Trainers: configures
README.md - Training data → Trainers: provides data
README.md - Trainers → Rollout engines: requests samples
README.md - Trainers → Reward functions: scores samples
README.md - Trainers → RL algorithms: computes updates
README.md - Trainers → Weight synchronization: syncs weights
README.md - Trainers → SFT training: runs SFT
README.md - Rollout engines → Model pipelines: generates with
README.md - Rollout engines → Sample format: fills request
unirl/rollout/engine/base.py - Model pipelines → Sample format: consumes
unirl/models/types/pipeline.py - BAGEL pipeline → WAN model bundle: model example
- Rollout engines → Worker dispatch: dispatches work
unirl/rollout/engine/base.py - Worker dispatch → Remote workers: uses handles
unirl/distributed/group/dispatch.py - Worker dispatch → Tensor batches: chunks batches
unirl/distributed/group/dispatch.py - Agentic rollout → Rollout engines: implements engine
unirl/rollout/engine/agentic/engine.py - Composed rollout → Rollout engines: implements engine
unirl/rollout/engine/composed/engine.py - Model pipelines → SDE kernels: uses transitions
- Sampling parameters → Sample format: configures generation
unirl/types/sample.py - Reward functions → Reward service: can use service
README.md