How microsoft/agentrl works
AgentRL standardizes tool-agent rollouts as trajectories, derives preference or reward-group training data, converts it to TRL datasets, and evaluates policies on task success, efficiency, and cost. The graph follows the documented workflow and sampled call behavior; some evaluation and environment internals are only partially sampled, so uncertain wiring is omitted.
Main entry points: environment.py Tool environment · evaluation.py Evaluation harness · collectors.py Rollout collectors · trajectory.py Trajectory model
Components 17
Rollout runtime
- Rollout collectors
agentrl/collectors.py - Tool environment
agentrl/environment.py - Tool interface
agentrl/tool.py
Trajectory data
- Trajectory model
agentrl/trajectory.py
Training preparation
- DPO pairing
agentrl/pairing.py - GRPO grouping
agentrl/grouping.py - Dataset converters
agentrl/converters.py
Evaluation
- Evaluation harness
agentrl/evaluation.py - Evaluation metrics
agentrl/metrics.py
Reference environments
- Toy environment
environments/toy/__init__.py - Office simulation
environments/office_sim/__init__.py - Toy task state
environments/toy/__init__.py - Office documents
environments/office_sim/__init__.py
Other components
- Engineer
- Agent policy
- TRL trainers
- Task definitions
Connections 23
- Engineer → Task definitions: defines tasks
- Engineer → Agent policy: supplies policy
- Engineer → Rollout collectors: starts collection
- Task definitions → Rollout collectors: provides tasks
- Agent policy → Rollout collectors: chooses actions
- Rollout collectors → Tool environment: runs episodes
- Tool environment → Tool interface: executes calls
- Tool environment → Trajectory model: records steps
- Toy environment → Tool environment: extends
- Office simulation → Tool environment: extends
- Toy environment → Toy task state: updates state
- Office simulation → Office documents: updates state
- Trajectory model → DPO pairing: supplies rollouts
- Trajectory model → GRPO grouping: supplies rollouts
- Trajectory model → Dataset converters: converts SFT/PPO
- DPO pairing → Dataset converters: converts DPO
- GRPO grouping → Dataset converters: converts GRPO
- Dataset converters → TRL trainers: provides datasets
- Engineer → Evaluation harness: starts evaluation
- Agent policy → Evaluation harness: provides policy
- Task definitions → Evaluation harness: provides tasks
- Evaluation harness → Rollout collectors: runs rollouts
- Evaluation harness → Evaluation metrics: computes metrics