How nvidia-nemo/gym works
NeMo Gym is a platform for evaluating and improving models and agents through stateful, verifiable task environments. The main workflow is to launch an evaluation, pair an agent harness and model with task data and an environment, execute rollouts (optionally in sandboxes and at scale), then score and inspect results. Separate capabilities include environment discovery and authoring, training integrations, rollout verification and comparison, observability, and export. The README establishes these product areas; sampled source excerpts are partial, so uncertain internal call wiring is not shown.
Main entry points: api.py Run orchestration · rollout_collection.py Rollout records · responses_api_agents Agent harnesses · main.py Gym CLI
Components 21
Run orchestration
- Gym CLI
nemo_gym/cli/main.py - Run orchestration
nemo_gym/orchestration/api.py - Job execution
nemo_gym/orchestration/jobs.py - Dataset loading
nemo_gym/dataset_orchestrator.py
Agent runtime
- Agent harnesses
responses_api_agents - Model server
responses_api_models/vllm_model/app.py - Sandbox providers
nemo_gym/sandbox/providers
Environment hub
- Environment server
nemo_gym/base_environment_server.py - Resource servers
resources_servers - Environment catalog
environments - Task verifier
nemo_gym/environment/_verifier_runner.py
Evaluation results
- Rollout records
nemo_gym/rollout_collection.py - Rollout health
nemo_gym/rollout_health.py - Reward reverification
nemo_gym/rollout_reverification.py - Run comparison
nemo_gym/comparison/runner.py - Result exporters
nemo_gym/exporters/base.py
Platform services
- Telemetry
nemo_gym/telemetry/setup.py - Agent and model registries
nemo_gym/agent_registry.py
Other components
- Evaluator
- Hosted inference
- Training framework
Connections 20
- Evaluator → Gym CLI: starts run
- Gym CLI → Run orchestration: submits run
- Run orchestration → Job execution: dispatches jobs
- Run orchestration → Dataset loading: loads tasks
- Run orchestration → Agent harnesses: dispatches rollouts
- Agent harnesses → Model server: requests inference
- Model server → Hosted inference: forwards inference
- Agent harnesses → Environment server: interacts with
- Environment server → Resource servers: serves resources
- Resource servers → Task verifier: scores task
- Agent harnesses → Sandbox providers: executes in
- Environment catalog → Environment server: configures tasks
- Run orchestration → Rollout records: collects results
- Rollout records → Rollout health: checks rollouts
- Rollout records → Reward reverification: recomputes rewards
- Rollout records → Run comparison: compares runs
- Rollout records → Result exporters: exports results
- Run orchestration → Telemetry: emits telemetry
- Gym CLI → Agent and model registries: resolves components
- Rollout records → Training framework: supplies training data