# microsoft/agentrl architecture

> Architecture diagram of the GitHub repository microsoft/agentrl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/agentrl
- Repository: https://github.com/microsoft/agentrl

## Overview

AgentRL standardizes tool-agent rollouts as trajectories, derives preference or reward-group training data, converts it to TRL datasets, and evaluates policies on task success, efficiency, and cost. The graph follows the documented workflow and sampled call behavior; some evaluation and environment internals are only partially sampled, so uncertain wiring is omitted.

Main entry points: [`agentrl/environment.py`](https://github.com/microsoft/agentrl/blob/main/agentrl/environment.py) (Tool environment), [`agentrl/evaluation.py`](https://github.com/microsoft/agentrl/blob/main/agentrl/evaluation.py) (Evaluation harness), [`agentrl/collectors.py`](https://github.com/microsoft/agentrl/blob/main/agentrl/collectors.py) (Rollout collectors), [`agentrl/trajectory.py`](https://github.com/microsoft/agentrl/blob/main/agentrl/trajectory.py) (Trajectory model)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Rollout runtime"]
  node_collectors["Rollout collectors<br/>[collectors.py]"]
  node_environment["Tool environment<br/>[environment.py]"]
  node_tools["Tool interface<br/>[tool.py]"]
end

subgraph group_data["Trajectory data"]
  node_trajectory["Trajectory model<br/>[trajectory.py]"]
end

subgraph group_training["Training preparation"]
  node_pairing["DPO pairing<br/>[pairing.py]"]
  node_grouping["GRPO grouping<br/>[grouping.py]"]
  node_converters["Dataset converters<br/>[converters.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_evaluation["Evaluation harness<br/>[evaluation.py]"]
  node_metrics["Evaluation metrics<br/>[metrics.py]"]
end

subgraph group_envs["Reference environments"]
  node_toy["Toy environment<br/>[__init__.py]"]
  node_office["Office simulation<br/>[__init__.py]"]
  node_toy_state[("Toy task state<br/>[__init__.py]")]
  node_office_state[("Office documents<br/>[__init__.py]")]
end

node_engineer(("Engineer"))
node_policy(("Agent policy"))
node_trl["TRL trainers"]
node_tasks["Task definitions"]

node_engineer -->|"defines tasks"| node_tasks
node_engineer -->|"supplies policy"| node_policy
node_engineer -->|"starts collection"| node_collectors
node_tasks -->|"provides tasks"| node_collectors
node_policy -->|"chooses actions"| node_collectors
node_collectors -->|"runs episodes"| node_environment
node_environment -->|"executes calls"| node_tools
node_environment -->|"records steps"| node_trajectory
node_toy -->|"extends"| node_environment
node_office -->|"extends"| node_environment
node_toy -->|"updates state"| node_toy_state
node_office -->|"updates state"| node_office_state
node_trajectory -->|"supplies rollouts"| node_pairing
node_trajectory -->|"supplies rollouts"| node_grouping
node_trajectory -->|"converts SFT/PPO"| node_converters
node_pairing -->|"converts DPO"| node_converters
node_grouping -->|"converts GRPO"| node_converters
node_converters -->|"provides datasets"| node_trl
node_engineer -->|"starts evaluation"| node_evaluation
node_policy -->|"provides policy"| node_evaluation
node_tasks -->|"provides tasks"| node_evaluation
node_evaluation -->|"runs rollouts"| node_collectors
node_evaluation -->|"computes metrics"| node_metrics

click node_collectors "https://github.com/microsoft/agentrl/blob/main/agentrl/collectors.py"
click node_environment "https://github.com/microsoft/agentrl/blob/main/agentrl/environment.py"
click node_tools "https://github.com/microsoft/agentrl/blob/main/agentrl/tool.py"
click node_trajectory "https://github.com/microsoft/agentrl/blob/main/agentrl/trajectory.py"
click node_pairing "https://github.com/microsoft/agentrl/blob/main/agentrl/pairing.py"
click node_grouping "https://github.com/microsoft/agentrl/blob/main/agentrl/grouping.py"
click node_converters "https://github.com/microsoft/agentrl/blob/main/agentrl/converters.py"
click node_evaluation "https://github.com/microsoft/agentrl/blob/main/agentrl/evaluation.py"
click node_metrics "https://github.com/microsoft/agentrl/blob/main/agentrl/metrics.py"
click node_toy "https://github.com/microsoft/agentrl/blob/main/environments/toy/__init__.py"
click node_office "https://github.com/microsoft/agentrl/blob/main/environments/office_sim/__init__.py"
click node_toy_state "https://github.com/microsoft/agentrl/blob/main/environments/toy/__init__.py"
click node_office_state "https://github.com/microsoft/agentrl/blob/main/environments/office_sim/__init__.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_collectors,node_environment,node_tools toneBlue
class node_trajectory toneAmber
class node_pairing,node_grouping,node_converters toneMint
class node_evaluation,node_metrics,node_tasks toneRose
class node_toy,node_office,node_toy_state,node_office_state,node_engineer,node_policy,node_trl toneIndigo
```

## Components

### Rollout runtime

- [Rollout collectors](https://github.com/microsoft/agentrl/blob/main/agentrl/collectors.py): `agentrl/collectors.py`
- [Tool environment](https://github.com/microsoft/agentrl/blob/main/agentrl/environment.py): `agentrl/environment.py`
- [Tool interface](https://github.com/microsoft/agentrl/blob/main/agentrl/tool.py): `agentrl/tool.py`

### Trajectory data

- [Trajectory model](https://github.com/microsoft/agentrl/blob/main/agentrl/trajectory.py): `agentrl/trajectory.py`

### Training preparation

- [DPO pairing](https://github.com/microsoft/agentrl/blob/main/agentrl/pairing.py): `agentrl/pairing.py`
- [GRPO grouping](https://github.com/microsoft/agentrl/blob/main/agentrl/grouping.py): `agentrl/grouping.py`
- [Dataset converters](https://github.com/microsoft/agentrl/blob/main/agentrl/converters.py): `agentrl/converters.py`

### Evaluation

- [Evaluation harness](https://github.com/microsoft/agentrl/blob/main/agentrl/evaluation.py): `agentrl/evaluation.py`
- [Evaluation metrics](https://github.com/microsoft/agentrl/blob/main/agentrl/metrics.py): `agentrl/metrics.py`

### Reference environments

- [Toy environment](https://github.com/microsoft/agentrl/blob/main/environments/toy/__init__.py): `environments/toy/__init__.py`
- [Office simulation](https://github.com/microsoft/agentrl/blob/main/environments/office_sim/__init__.py): `environments/office_sim/__init__.py`
- [Toy task state](https://github.com/microsoft/agentrl/blob/main/environments/toy/__init__.py): `environments/toy/__init__.py`
- [Office documents](https://github.com/microsoft/agentrl/blob/main/environments/office_sim/__init__.py): `environments/office_sim/__init__.py`

### Other components

- **Engineer**
- **Agent policy**
- **TRL trainers**
- **Task definitions**

## Connections

- Engineer → Task definitions: defines tasks
- Engineer → Agent policy: supplies policy
- Engineer → Rollout collectors: starts collection
- Task definitions → Rollout collectors: provides tasks
- Agent policy → Rollout collectors: chooses actions
- Rollout collectors → Tool environment: runs episodes
- Tool environment → Tool interface: executes calls
- Tool environment → Trajectory model: records steps
- Toy environment → Tool environment: extends
- Office simulation → Tool environment: extends
- Toy environment → Toy task state: updates state
- Office simulation → Office documents: updates state
- Trajectory model → DPO pairing: supplies rollouts
- Trajectory model → GRPO grouping: supplies rollouts
- Trajectory model → Dataset converters: converts SFT/PPO
- DPO pairing → Dataset converters: converts DPO
- GRPO grouping → Dataset converters: converts GRPO
- Dataset converters → TRL trainers: provides datasets
- Engineer → Evaluation harness: starts evaluation
- Agent policy → Evaluation harness: provides policy
- Task definitions → Evaluation harness: provides tasks
- Evaluation harness → Rollout collectors: runs rollouts
- Evaluation harness → Evaluation metrics: computes metrics