How nvidia-nemo/prorl-agent-server works
1. What the repo is
Polar is a rollout-as-a-service system for agentic RL. Its core architecture separates task submission, distributed execution, trajectory construction, and evaluation so training frameworks can consume rollouts without embedding agent harness logic directly.
2. Top-level request flow
A client submits a rollout task through the rollout service. The rollout server acts as the central orchestrator, then assigns work to one or more gateway nodes. Gateways run the agent work asynchronously, collect outputs, and return structured rollout artifacts for later consumption.
3. Rollout orchestration layer
src/polar/rollout contains the scheduling and control plane: balancing, pipeline management, timing, and the rollout server entrypoint. This layer is responsible for admission, dispatch, and coordinating many in-flight rollouts rather than executing the agent itself.
4. Gateway execution layer
src/polar/gateway is the per-node runtime boundary. It includes node/session management, request proxying, dispatch, storage, and completion writing. This is where the system bridges between “task received” and “agent actually ran,” while keeping execution state isolated per node.
5. Agent harness abstraction
src/polar/agent defines the harness contract and a set of presets for common agent styles such as shell-based agents, Claude Code, Codex, Gemini CLI, OpenHands, Qwen Code, and others. This makes agent execution backend-agnostic: Polar cares about the harness interface, not the specific agent implementation.
6. Runtime isolation
src/polar/runtime abstracts how harnesses are launched, with support for Docker and Apptainer. Example directories include runtime Dockerfiles and image-build scripts, showing that Polar expects agent environments to be containerized and reproducible across different harnesses and benchmarks.
7. Inference-server boundary
The repo explicitly supports both SGLang and vLLM via alternate topology files and patch scripts. Polar does not hardwire an inference engine; instead, gateways and agent harnesses communicate through provider-style HTTP boundaries so training can target whichever backend the harness supports.
8. API transformation and proxying
src/polar/gateway/transform normalizes different model-provider request/response shapes, including OpenAI chat, OpenAI responses, Anthropic, Google, images, and reasoning metadata. This layer is important because it decouples agent code from a specific LLM API schema.
9. Trajectory construction and evaluation
src/polar/trajectory turns raw execution traces into training-ready trajectories. Builders handle how records are merged or assembled per request, while evaluators score or validate outputs using strategies like session completion, test-on-output, SWE-bench harness integration, and other registered evaluators.
10. Platform services and observability
src/polar/platform provides task/session/event APIs, topology exposure, an SSE fanout mechanism, and filesystem indexing for stored artifacts. This is the operational control plane that supports the dashboard and external tooling, separate from the execution pipeline itself.
11. Trainer integration: Slime bridge
src/slime_bridge is the demo RL/training integration layer. It adapts Polar rollout outputs into training-time messages, reward computation, rollout acceptance logic, and post-processing, showing how Polar plugs into an external RL stack without owning the trainer.
12. Examples as architectural blueprints
The examples/ tree shows the expected deployment pattern end-to-end: each example has a task runner, a runtime image, and one or more topology files. The examples span a calculator smoke test, count-stars vision task, SWE-bench Verified evaluation, Tmax-15k, and a SWE-Gym/GRPO training path, demonstrating that the architecture is meant to generalize across benchmark, agentic coding, and training workloads.
13. UI and operator boundary
web/ is an optional React/Vite dashboard for monitoring tasks, sessions, traces, completions, topology, and reward curves. It sits on top of the platform APIs and SSE streams; it does not participate in rollout execution, only observation and comparison.
14. Configuration model
The repo’s deployment unit is a shared topology file. The same topology drives polar serve_rollout, polar serve_gateway, polar submit, polar status, and the dashboard, which makes the system diagram naturally centered around a single configuration graph connecting orchestrator, gateways, runtimes, and inference backends.
15. Key architectural boundary
The main design principle is that agent logic, inference serving, runtime isolation, rollout orchestration, and evaluation are all separate layers. That separation is what allows Polar to support many harnesses and trainers while keeping the rollout server itself neutral and scalable.
Main entry points: node.py Node runtime · topology.py Topology · rollout/server.py Rollout server · platform/server.py Platform server
Components 25
Control plane
Submission, orchestration, and platform APIs
- Topology · Shared topology model that wires rollout, gateways, runtimes, and backends
src/polar/config/topology.py - Rollout server · Entry point for task admission, dispatch, and rollout coordination
src/polar/rollout/server.py - Manager · Coordinates in-flight rollouts and overall control flow
src/polar/rollout/manager.py - Balancer · Assigns work across gateway nodes and load-balances rollout tasks
src/polar/rollout/balancer.py - Platform server · Operational API surface for tasks, sessions, events, and topology
src/polar/platform/server.py - Task API · Task and session endpoints used by tools and UI
src/polar/platform/api/tasks.py - SSE fanout · Pushes live task and session updates to observers
src/polar/platform/sse_fanout.py
Execution
Gateway runtime, agent harnesses, and model API adaptation
- Gateway server · Per-node execution boundary that receives assigned rollout work
src/polar/gateway/server.py - Node runtime · Maintains node and session state for isolated execution
src/polar/gateway/node.py - Dispatcher · Routes requests into the active agent execution path
src/polar/gateway/dispatcher.py - Proxy · Bridges rollout traffic to provider-style model endpoints
src/polar/gateway/proxy.py - API transforms · Normalizes OpenAI, Anthropic, Google, image, and reasoning payloads
src/polar/gateway/transform/base.py - Agent factory · Builds agent harnesses from presets and execution models
src/polar/agent/factory.py - Runtime factory · Creates containerized launchers for harness execution
src/polar/runtime/factory.py - Docker · Docker-based runtime isolation for harnesses
src/polar/runtime/docker.py - Apptainer · Apptainer-based runtime isolation for harnesses
src/polar/runtime/apptainer.py
Trajectories
Turn rollout artifacts into training-ready outputs and scores
- Trajectory registry · Registers builders and evaluators for trajectory assembly
src/polar/trajectory/registry.py - Builders · Combines raw records into training-ready trajectories
src/polar/trajectory/builder/base.py - Evaluators · Scores or validates rollout outputs with pluggable strategies
src/polar/trajectory/evaluator/base.py
Training integration
Bridge into external RL/training flows
- Slime adapter · Adapts Polar rollout outputs into trainer-facing messages and logic
src/slime_bridge/adapter.py - Reward · Computes reward signals and post-processing for training
src/slime_bridge/reward.py
Examples and UI
Deployment blueprints and the operator dashboard
- Example topologies · Example rollout layouts for benchmarks and training stacks
examples/calculator/topology.sgl.yaml - Example runners · Scripts that submit benchmark or training tasks through Polar
examples/calculator/run.py - Dashboard · Operator UI for monitoring tasks, sessions, traces, and rewards
web/src/routes/Dashboard.tsx - API client · Frontend client for platform data and SSE streams
web/src/api/client.ts
Connections 27
- Topology → Rollout server: configure · Topology drives rollout orchestration
- Topology → Gateway server: configure · Topology selects gateway nodes and backends
- Topology → Platform server: configure · Platform exposes the same deployment graph
- Rollout server → Manager: coordinate · Server delegates rollout lifecycle control
- Manager → Balancer: schedule · Manager assigns work through balancing logic
- Rollout server → Gateway server: dispatch · Rollout work is sent to gateway nodes
- Gateway server → Node runtime: host · Gateway keeps per-node state isolated
- Node runtime → Dispatcher: route · Node runtime sends requests into dispatch
- Dispatcher → Proxy: proxy · Dispatcher forwards provider-bound traffic
- Proxy → API transforms: normalize · Proxy normalizes request and response schemas
- Node runtime → Agent factory: build · Node instantiates the selected agent harness
- Agent factory → Runtime factory: launch · Harness execution is bound to a runtime launcher
- Runtime factory → Docker: backend · Docker is one supported isolated runtime
- Runtime factory → Apptainer: backend · Apptainer is the other supported isolated runtime
- Node runtime → Builders: emit · Execution outputs become trajectory records
- Trajectory registry → Builders: register · Registry provides builder implementations
- Trajectory registry → Evaluators: register · Registry provides evaluator implementations
- Builders → Evaluators: score · Built trajectories are passed into evaluation
- Slime adapter → Rollout server: consume · Training integration consumes rollout artifacts
- Slime adapter → Reward: reward · Adapter computes trainer-facing reward logic
- Platform server → Task API: serve · Platform exposes task and session APIs
- Platform server → SSE fanout: stream · Platform publishes live updates to observers
- Dashboard → API client: query · Dashboard uses the shared API client
- API client → Platform server: read · UI reads tasks, sessions, and topology from platform APIs
- API client → SSE fanout: subscribe · UI subscribes to live event streams
- Example topologies → Example runners: drive · Examples pair topology files with submission scripts
- Example runners → Topology: mirror · Examples demonstrate the same deployment model