# primeintellect-ai/verifiers architecture

> Architecture diagram of the GitHub repository primeintellect-ai/verifiers, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/primeintellect-ai/verifiers
- Repository: https://github.com/primeintellect-ai/verifiers
- Video tour (about a minute): https://gitdiagram.com/primeintellect-ai/verifiers/video

## Overview

This repository is a Python platform for building, serving, evaluating, and training on LLM “environments.” The architecture is centered on a core `verifiers` library, a large catalog of example environments, and supporting CLI/training/serve layers that connect environments to model endpoints and RL workflows.

#### High-level structure

The repo is organized into four main planes:

1. **Core runtime library** (`verifiers/`)
   - Defines the environment abstractions, rubrics, parsers, model clients, and shared utilities.
   - Provides the CLI entrypoints used by developers and automated workflows.

2. **Environment implementations** (`environments/`)
   - Contains self-contained tasks and benchmark environments.
   - Each environment is typically a Python module with its own `pyproject.toml` and `README`.
   - Some are simple single-file tasks; others are multi-file projects with Dockerized components, browser integrations, or OpenEnv/TextArena-style client/server layouts.

3. **RL/inference stack** (`verifiers/rl/` and `packages/verifiers-rl/`)
   - Implements training orchestration and inference-server/client support for reinforcement-learning usage.
   - This portion is split out as a reusable package boundary, likely to support separate distribution and dependency management.

4. **Operational tooling and docs**
   - GitHub Actions for testing, style, publish, and release automation.
   - Docs, release notes, templates, and setup assets for workspace initialization.

#### Core architecture: how an environment works

The fundamental unit is an **environment module** that exposes a `load_environment` function returning a `verifiers.Environment` instance.

That environment object typically combines:
- a **dataset** of prompts/tasks,
- a **harness** for model interaction,
- and a **rubric/reward function** to score outputs.

From an architecture perspective, the flow is:

- task data enters from a dataset or task generator,
- a model interacts through the environment harness,
- the environment produces observations, tool results, or state transitions,
- a rubric computes reward/metrics,
- results are saved, displayed, or fed into RL.

This is the central data flow used both for evaluation and training.

#### Main subsystems in `verifiers/`

##### 1. Environment abstraction layer
Located under `verifiers/envs/`, this is the core type system and implementation boundary for environments.

Important pieces:
- `environment.py`, `singleturn_env.py`, `multiturn_env.py`, `tool_env.py`, `stateful_tool_env.py`, `sandbox_env.py`, `python_env.py`
- `env_group.py` for grouping or composing multiple environments
- `experimental/` for specialized runtime patterns:
  - browser/DOM/CUA environments
  - CLI-agent environments
  - gym-style environments
  - OpenEnv, MCP, Harbor, RLM, QA variants
  - sandbox mixins for isolation and controlled execution

This layer is the main architectural seam between “task definition” and “execution protocol.”

##### 2. Rubrics and scoring
Located under `verifiers/rubrics/`.

This subsystem turns environment trajectories into scalar signals or structured metrics.

Notable pieces:
- `rubric.py`, `rubric_group.py`
- task-specific rubrics like `math_rubric.py` and `judge_rubric.py`
- experimental rubric variants such as `hybrid_math_rubric.py`

Architecturally, rubrics are a separate scoring layer so that environment behavior and evaluation logic are decoupled.

##### 3. Parsing and response interpretation
Located under `verifiers/parsers/`.

This layer extracts structured meaning from model output:
- `think_parser.py`
- `maybe_think_parser.py`
- `xml_parser.py`

It sits between raw model completions and downstream evaluation or tool routing.

##### 4. Client stack for model APIs
Located under `verifiers/clients/`.

This is the abstraction for LLM endpoint access:
- OpenAI-style chat completions
- token-based variants
- Anthropic messages client
- completions client

This means the repo is endpoint-agnostic at the core, with a pluggable client layer for different API shapes.

##### 5. Serve layer for remote environment execution
Located under `verifiers/serve/`.

This subsystem is the process/network boundary for running environments outside the caller process.

Key elements:
- server side: `env_server.py`, `env_router.py`, `env_worker.py`, `zmq_env_server.py`
- client side: `env_client.py`, `zmq_env_client.py`

Architecturally, this creates a remote execution boundary:
- the caller talks to an environment client,
- the client forwards actions to a server/worker,
- workers execute the environment and return observations/results.

This is important for isolation, distributed execution, or sandboxed workloads.

##### 6. Utilities and infrastructure support
`verifiers/utils/` contains the supporting glue:
- config loading
- logging/display
- message and tool handling
- save/load helpers
- process/thread orchestration
- heartbeat and tunneling
- error chaining and interception
- eval display and metrics

These utilities support the main runtime but are not themselves the domain model.

#### CLI and user-facing workflows

The main CLI surface is under `verifiers/cli/` and `verifiers/scripts/`.

Commands cover the lifecycle:
- initialize a workspace or environment
- install environments
- build environments
- run evaluations
- run GEPA/prompt optimization flows
- run training/prime-rl-related workflows
- open a terminal UI for eval results

There is also a plugin layer, notably `verifiers/cli/plugins/prime.py`, which ties this repo into the Prime Intellect ecosystem.

Architecturally, the CLI is the orchestration entrypoint:
- it reads configs,
- loads environments,
- connects to model endpoints,
- launches runs,
- and writes results.

#### RL training and inference

There are two closely related RL code paths:
- `verifiers/rl/`
- `packages/verifiers-rl/verifiers_rl/rl/`

They contain:
- inference client/server
- trainer orchestration
- configuration
- trainer utilities

The split suggests a package boundary for RL-specific functionality separate from the main environment library.

The RL data flow is:
- environment emits trajectories and rewards,
- trainer/orchestrator coordinates rollouts and updates,
- inference service provides model execution,
- trajectories are used for token-by-token or turn-based RL training.

This matches the README’s focus on “environments for LLM reinforcement learning.”

#### Environment catalog under `environments/`

This directory contains the actual benchmark/task implementations. It is not just samples; it is the repository’s environment content layer.

There are several patterns:

##### Simple Python environments
Examples:
- `wordle`
- `reverse_text`
- `sentence_repeater`
- `gsm8k`
- `math_python`
- `wiki_search`
- `tool_test`
- `self_reward`

These are usually one main Python file plus metadata and docs.

##### Multi-turn or stateful environments
Examples:
- `doublecheck`
- `continuation_quality`
- `math_group`
- `mmmu`

These emphasize conversation history, tool usage, or composite scoring.

##### External-system integrations
Examples:
- `browser_cua_example`
- `browser_dom_example`
- `mcp_search_env`
- `openenv_echo`
- `openenv_textarena`
- `reasoning_gym_env`
- `terminus_harbor`
- `opencode_harbor`

These wrap external protocols, browser automation, game engines, or third-party environment formats.

##### Dockerized task packs
Especially in `opencode_harbor/tasks/` and `terminus_harbor/tasks/`, each task is packaged with:
- instructions
- environment Dockerfile
- solution artifact
- tests

This forms a benchmark/task sub-architecture where each task is isolated as a reproducible containerized unit.

#### External boundaries and integrations

The repo explicitly bridges several external systems:

- **OpenAI-compatible endpoints** via configurable API endpoints
- **Prime Intellect infrastructure** for hosted training, inference, and the environment hub
- **Browser automation** via DOM/CUA browser modes
- **OpenEnv / TextArena / ReasoningGym** style integrations
- **Docker** for task environments and sandboxed execution
- **ZMQ** for server/client environment transport in the serve layer

These are important diagram boundaries because much of the architecture is about adapting to heterogeneous environment and model runtimes.

#### Repository-level workflows

A few top-level artifacts matter architecturally:

- `.github/workflows/` handles style checks, tests, docs sync, environment publishing, and release tagging.
- `configs/` contains example evaluation, RL, and endpoint configuration files used by the CLI and trainers.
- `assets/` contains templates, release notes, docs redirection, and packaged workflow guidance for AI coding agents and environment authors.
- `tests/` is broad and cross-cuts the whole system, validating env abstractions, clients, parsers, rubrics, serve behavior, CLI commands, and integrations.

#### Suggested diagram boundaries

If you draw this repo as a system diagram, the most accurate boundaries are:

1. **Environment authoring boundary**
   - `environments/` task modules, examples, and Dockerized task packs

2. **Runtime/core boundary**
   - `verifiers/` environment abstractions, rubrics, parsers, clients, utilities

3. **Execution boundary**
   - `verifiers/serve/` for remote or isolated environment execution

4. **Model boundary**
   - external LLM endpoints accessed through client adapters

5. **Training/evaluation boundary**
   - CLI and RL trainer/inference layers that consume environment trajectories and emit metrics or policy updates

Overall, the repo is a modular environment platform: environments are first-class packages, evaluation is rubric-driven, model access is client-abstraction-based, and training can reuse the same environment contract through a separate RL orchestration stack.

Main entry points: [`verifiers/envs`](https://github.com/primeintellect-ai/verifiers/tree/main/verifiers/envs) (Env model), [`verifiers/cli`](https://github.com/primeintellect-ai/verifiers/tree/main/verifiers/cli) (CLI), [`verifiers/serve`](https://github.com/primeintellect-ai/verifiers/tree/main/verifiers/serve) (Serve), [`verifiers/rl`](https://github.com/primeintellect-ai/verifiers/tree/main/verifiers/rl) (RL stack)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core runtime"]
  node_verifiers_envs["Env model<br/>runtime abstractions"]
  node_verifiers_rubrics["Rubrics<br/>scoring"]
  node_verifiers_parsers["Parsers<br/>output parsing"]
  node_verifiers_clients["Clients<br/>llm adapters"]
  node_verifiers_serve["Serve<br/>remote exec"]
  node_verifiers_utils["Utils<br/>support library"]
end

subgraph group_platform["Platform layers"]
  node_verifiers_cli["CLI<br/>user entrypoints"]
  node_verifiers_scripts["Scripts"]
  node_verifiers_rl["RL stack<br/>training orchestration"]
  node_packages_verifiers_rl["RL package<br/>packaged rl boundary"]
end

subgraph group_catalog["Environment catalog"]
  node_environments_catalog["Environments<br/>task catalog"]
  node_openenv_echo["OpenEnv echo<br/>client-server env"]
  node_openenv_textarena["TextArena env<br/>client-server env"]
  node_opencode_harbor["OpenCode tasks<br/>dockerized tasks"]
  node_terminus_harbor["Terminus tasks<br/>dockerized tasks"]
  node_simple_envs["Simple tasks<br/>single-file envs"]
  node_browser_envs["Browser envs<br/>browser integrations"]
end

subgraph group_ops["Operations"]
  node_configs["Configs<br/>runtime config"]
  node_docs["Docs<br/>documentation"]
end

node_verifiers_cli -->|"loads"| node_verifiers_envs
node_verifiers_cli -->|"uses"| node_verifiers_clients
node_verifiers_cli -->|"evaluates"| node_verifiers_rubrics
node_verifiers_scripts -->|"wraps"| node_verifiers_cli
node_verifiers_envs -->|"interprets"| node_verifiers_parsers
node_verifiers_envs -->|"scores by"| node_verifiers_rubrics
node_verifiers_envs -->|"calls"| node_verifiers_clients
node_verifiers_serve -->|"executes"| node_verifiers_envs
node_verifiers_serve -->|"orchestrates"| node_verifiers_utils
node_verifiers_rl -->|"rolls out"| node_verifiers_envs
node_verifiers_rl -->|"deploys via"| node_verifiers_serve
node_packages_verifiers_rl -.->|"mirrors"| node_verifiers_rl
node_environments_catalog -->|"implements"| node_verifiers_envs
node_openenv_echo -->|"adapts"| node_verifiers_envs
node_openenv_textarena -->|"adapts"| node_verifiers_envs
node_opencode_harbor -->|"runs under"| node_verifiers_envs
node_terminus_harbor -->|"runs under"| node_verifiers_envs
node_simple_envs -->|"examples of"| node_verifiers_envs
node_browser_envs -->|"extends"| node_verifiers_envs
node_configs -->|"configures"| node_verifiers_cli
node_docs -.->|"documents"| node_verifiers_cli

click node_verifiers_envs "https://github.com/primeintellect-ai/verifiers/tree/main/verifiers/envs"
click node_verifiers_rubrics "https://github.com/primeintellect-ai/verifiers/tree/main/verifiers/rubrics"
click node_verifiers_parsers "https://github.com/primeintellect-ai/verifiers/tree/main/verifiers/parsers"
click node_verifiers_clients "https://github.com/primeintellect-ai/verifiers/tree/main/verifiers/clients"
click node_verifiers_serve "https://github.com/primeintellect-ai/verifiers/tree/main/verifiers/serve"
click node_verifiers_utils "https://github.com/primeintellect-ai/verifiers/tree/main/verifiers/utils"
click node_verifiers_cli "https://github.com/primeintellect-ai/verifiers/tree/main/verifiers/cli"
click node_verifiers_scripts "https://github.com/primeintellect-ai/verifiers/tree/main/verifiers/scripts"
click node_verifiers_rl "https://github.com/primeintellect-ai/verifiers/tree/main/verifiers/rl"
click node_packages_verifiers_rl "https://github.com/primeintellect-ai/verifiers/tree/main/packages/verifiers-rl/verifiers_rl/rl"
click node_environments_catalog "https://github.com/primeintellect-ai/verifiers/tree/main/environments"
click node_openenv_echo "https://github.com/primeintellect-ai/verifiers/tree/main/environments/openenv_echo"
click node_openenv_textarena "https://github.com/primeintellect-ai/verifiers/tree/main/environments/openenv_textarena"
click node_opencode_harbor "https://github.com/primeintellect-ai/verifiers/tree/main/environments/opencode_harbor"
click node_terminus_harbor "https://github.com/primeintellect-ai/verifiers/tree/main/environments/terminus_harbor"
click node_simple_envs "https://github.com/primeintellect-ai/verifiers/tree/main/environments/wordle"
click node_browser_envs "https://github.com/primeintellect-ai/verifiers/tree/main/environments/browser_cua_example"
click node_configs "https://github.com/primeintellect-ai/verifiers/tree/main/configs"
click node_docs "https://github.com/primeintellect-ai/verifiers/tree/main/docs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_verifiers_envs,node_verifiers_rubrics,node_verifiers_parsers,node_verifiers_clients,node_verifiers_serve,node_verifiers_utils toneBlue
class node_verifiers_cli,node_verifiers_scripts,node_verifiers_rl,node_packages_verifiers_rl toneAmber
class node_environments_catalog,node_openenv_echo,node_openenv_textarena,node_opencode_harbor,node_terminus_harbor,node_simple_envs,node_browser_envs toneMint
class node_configs,node_docs toneRose
```

## Components

### Core runtime

Shared library for environments, clients, rubrics, parsing, serving, and utilities.

- [Env model](https://github.com/primeintellect-ai/verifiers/tree/main/verifiers/envs): runtime abstractions · Core environment types and execution protocols. · `verifiers/envs`
- [Rubrics](https://github.com/primeintellect-ai/verifiers/tree/main/verifiers/rubrics): scoring · Reward and metric computation for trajectories. · `verifiers/rubrics`
- [Parsers](https://github.com/primeintellect-ai/verifiers/tree/main/verifiers/parsers): output parsing · Structured interpretation of model responses. · `verifiers/parsers`
- [Clients](https://github.com/primeintellect-ai/verifiers/tree/main/verifiers/clients): llm adapters · Provider-neutral model API clients. · `verifiers/clients`
- [Serve](https://github.com/primeintellect-ai/verifiers/tree/main/verifiers/serve): remote exec · Client/server boundary for isolated environment execution. · `verifiers/serve`
- [Utils](https://github.com/primeintellect-ai/verifiers/tree/main/verifiers/utils): support library · Shared glue for config, messaging, logging, saving, and orchestration. · `verifiers/utils`

### Platform layers

User-facing CLI, scripts, and RL orchestration built on top of the core library.

- [CLI](https://github.com/primeintellect-ai/verifiers/tree/main/verifiers/cli): user entrypoints · Command-line orchestration for init, build, eval, setup, and tuning flows. · `verifiers/cli`
- [Scripts](https://github.com/primeintellect-ai/verifiers/tree/main/verifiers/scripts): entry scripts · Top-level script wrappers for common workflows. · `verifiers/scripts`
- [RL stack](https://github.com/primeintellect-ai/verifiers/tree/main/verifiers/rl): training orchestration · Trainer and inference services for RL workflows. · `verifiers/rl`
- [RL package](https://github.com/primeintellect-ai/verifiers/tree/main/packages/verifiers-rl/verifiers_rl/rl): packaged rl boundary · Separate distributable RL package mirror. · `packages/verifiers-rl/verifiers_rl/rl`

### Environment catalog

Repository of benchmark and example environment packages.

- [Environments](https://github.com/primeintellect-ai/verifiers/tree/main/environments): task catalog · Collection of benchmark and example environment packages. · `environments`
- [OpenEnv echo](https://github.com/primeintellect-ai/verifiers/tree/main/environments/openenv_echo): client-server env · OpenEnv-style example environment with server/client split. · `environments/openenv_echo`
- [TextArena env](https://github.com/primeintellect-ai/verifiers/tree/main/environments/openenv_textarena): client-server env · OpenEnv/TextArena integration with rewards and server runtime. · `environments/openenv_textarena`
- [OpenCode tasks](https://github.com/primeintellect-ai/verifiers/tree/main/environments/opencode_harbor): dockerized tasks · Containerized benchmark tasks with instructions, solutions, and tests. · `environments/opencode_harbor`
- [Terminus tasks](https://github.com/primeintellect-ai/verifiers/tree/main/environments/terminus_harbor): dockerized tasks · Containerized task pack for harbor-style environments. · `environments/terminus_harbor`
- [Simple tasks](https://github.com/primeintellect-ai/verifiers/tree/main/environments/wordle): single-file envs · Representative single-file environments like wordle, gsm8k, math, and wiki search. · `environments/wordle`
- [Browser envs](https://github.com/primeintellect-ai/verifiers/tree/main/environments/browser_cua_example): browser integrations · Browser CUA and DOM example environments. · `environments/browser_cua_example`

### Operations

Docs, configs, templates, and release automation.

- [Configs](https://github.com/primeintellect-ai/verifiers/tree/main/configs): runtime config · Evaluation, RL, and endpoint configuration files. · `configs`
- [Docs](https://github.com/primeintellect-ai/verifiers/tree/main/docs): documentation · Project documentation and reference guides. · `docs`

## Connections

- CLI → Env model: loads · CLI resolves environment definitions and execution modes.
- CLI → Clients: uses · CLI connects runs to model endpoints.
- CLI → Rubrics: evaluates · CLI-driven evaluation and tuning uses rubric scoring.
- Scripts → CLI: wraps · Top-level scripts delegate into CLI commands.
- Env model → Parsers: interprets · Environment protocols rely on parsed model output.
- Env model → Rubrics: scores by · Trajectory scoring is separated into rubrics.
- Env model → Clients: calls · Environments interact with model backends through client adapters.
- Serve → Env model: executes · Serve layer runs environments across process or network boundaries.
- Serve → Utils: orchestrates · Serve uses shared process, heartbeat, and transport utilities.
- RL stack → Env model: rolls out · Training stack consumes environment trajectories and rewards.
- RL stack → Serve: deploys via · RL can use remote environment execution.
- RL package → RL stack: mirrors · Reusable package boundary for RL functionality.
- Environments → Env model: implements · Environment packages build on the core environment abstractions.
- OpenEnv echo → Env model: adapts · Client/server OpenEnv example built on environment contracts.
- TextArena env → Env model: adapts · TextArena integration plugs into the environment layer.
- OpenCode tasks → Env model: runs under · Dockerized tasks are exposed as environment packages.
- Terminus tasks → Env model: runs under · Harbor task pack follows the same environment contract.
- Simple tasks → Env model: examples of · Representative single-file benchmark environments.
- Browser envs → Env model: extends · Browser modes use specialized experimental environment types.
- Configs → CLI: configures · Evaluation and RL configs feed the CLI and trainers.
- Docs → CLI: documents · Documentation explains setup, evaluation, and training flows.
