How nvidia/isaac-gr00t works
1. What this repository is
- This is the reference implementation for NVIDIA Isaac GR00T N1.7, a vision-language-action robot policy stack.
- The core runtime is a Python package (gr00t) plus scripts, examples, and benchmark-specific adapters for training, evaluation, and deployment.
2. Primary system boundary
- The repo splits cleanly into two planes:
- Offline plane: data preparation, dataset loading, training/fine-tuning, checkpoint export.
- Online plane: policy serving, open-loop evaluation, closed-loop sim/real-robot control.
- Most architecture diagrams should show data flowing from robot demos into a normalized dataset, then into model training, then into a policy server or inference script.
3. Data layer
- The data format is a LeRobot v2-style dataset with GR00T-specific meta/modality.json.
- Each dataset contains:
- meta/ for episode index, tasks, stats, and modality mapping
- data/ parquet files for state/action sequences
- videos/ mp4 recordings per observation stream
- optional masks/ for mask-guided experiments
- This layer is where embodiment-specific state/action layouts are defined and interpreted.
4. Dataset ingestion and normalization
- gr00t/data/ handles dataset loading, sharded dataset variants, collators, statistics, and embodiment tags.
- state_action/ contains the logic for action representation, including relative end-effector processing and chunking.
- The key architectural boundary here is between raw demonstration files and the model-ready tensors produced by the dataset/collator pipeline.
5. Embodiment abstraction
- gr00t/data/embodiment_tags.py and gr00t/configs/data/embodiment_configs.py define the bridge between a robot/environment and the model’s expected modalities.
- Embodiment tags select the state keys, action semantics, normalization, and video keys used at inference and training time.
- This is the main mechanism that allows one model family to support DROID, LIBERO, SimplerEnv, and custom robots like SO100.
6. Model architecture
- The model code lives under gr00t/model/.
- The top-level GR00T N1.7 model combines:
- a vision-language backbone (qwen3_backbone.py / GR00T N1.7 processing)
- a diffusion/flow-matching action head for continuous action generation
- model/base/ and model/modules/ separate reusable model pipeline pieces from architecture-specific components.
7. Training and fine-tuning pipeline
- gr00t/experiment/ is the training orchestration layer.
- launch_train.py and launch_finetune.py are the main entry points; trainer.py encapsulates distributed training behavior.
- Configs under gr00t/configs/ define model, data, training, and fine-tuning settings.
- The training path is designed for single-GPU and multi-GPU launches, including DeepSpeed configuration for scale-out fine-tuning.
8. Policy and inference boundary
- gr00t/policy/ turns a trained model into a runtime policy interface.
- Gr00tPolicy is the main abstraction used by both standalone inference and the server-client deployment path.
- replay_policy.py supports debugging by replaying dataset actions instead of using a learned model.
9. Serving architecture for deployment
- gr00t/eval/run_gr00t_server.py exposes the policy over ZMQ as a lightweight action-serving process.
- Clients send observations and receive actions, which keeps the heavy model isolated on a GPU host.
- This separation is important for real robots and distributed evaluation: the control loop can stay near the robot while inference runs elsewhere.
10. Evaluation stack
- gr00t/eval/open_loop_eval.py compares model-predicted actions against recorded trajectories and generates visual diagnostics.
- gr00t/eval/sim/ contains simulation adapters, wrappers, and environment utilities for benchmark execution.
- gr00t/eval/real_robot/ contains real-robot evaluation entry points, such as SO100-specific evaluation code.
- The evaluation system is built around the same policy interface, so simulation and hardware can share the same inference contract.
11. Benchmark and example integrations
- examples/ contains reproducible benchmark workflows for DROID, LIBERO, SimplerEnv, SO100, and mask-guided background suppression.
- These folders typically define benchmark-specific modality configs, dataset conversion helpers, and runnable evaluation/fine-tuning commands.
- external_dependencies/ holds benchmark repos that are treated as integration dependencies rather than core code.
12. Deployment and acceleration toolchain
- scripts/deployment/ contains export and acceleration paths:
- ONNX export
- TensorRT engine building
- TRT pipeline wiring
- platform-specific setup for dGPU, Orin, Thor, and Spark
- This is the inference-optimization boundary: model code remains in Python, while deployment scripts bridge to platform-specific runtimes and GPU acceleration stacks.
13. Video and media handling
- gr00t/utils/video_utils.py and related utilities support dataset video decoding and processing.
- The repo explicitly supports different video backends depending on platform, with torchcodec emphasized for aarch64 systems.
- This matters architecturally because visual observations are a first-class input to the policy, not an auxiliary asset.
14. Tooling, packaging, and CI
- The repo uses uv for environment and dependency management, with platform-specific Dockerfiles under docker/ and CI in .github/workflows.
- Tests cover datasets, model forward paths, policy APIs, evaluation utilities, deployment scripts, and example workflows.
- The test and CI structure mirrors the runtime architecture, which makes it easier to validate each boundary independently.
15. How to draw the main data flow
- A good diagram would show:
- raw robot/video demonstrations → dataset conversion/normalization → sharded LeRobot-style datasets
- datasets + modality config + embodiment tag → model training/fine-tuning
- trained checkpoint → standalone inference or ZMQ policy server
- policy server → simulation client or real robot controller
- evaluation outputs → metrics/plots/benchmark results
- The central concept is that embodiment tags and modality configs connect data, model, and deployment into one consistent control interface.
Main entry points: gr00t_n1d7.py GR00T N1.7 · gr00t_policy.py Gr00tPolicy · run_gr00t_server.py ZMQ server · factory.py Dataset loader
Components 23
Data plane
Dataset ingestion, normalization, and embodiment mapping.
- Robot demos · Raw robot demonstrations and recorded videos before normalization.
demo_data - LeRobot dataset · GR00T-style episode, meta, video, and optional mask layout.
demo_data/cube_to_bowl_5 - Dataset loader · Loads episodes, sharded datasets, and dataset variants.
gr00t/data/dataset/factory.py - State actions · Action representation, chunking, and pose/state conversion.
gr00t/data/state_action/state_action_processor.py - Embodiment tags · Maps robot-specific state, action, and video keys into a shared interface.
gr00t/data/embodiment_tags.py - Embodiment configs · Config layer that selects modality and normalization rules per robot family.
gr00t/configs/data/embodiment_configs.py - Collators · Builds model-ready batches and normalized tensors from dataset samples.
gr00t/data/collator/collators.py - Stats & video · Dataset statistics and video decoding/processing utilities.
gr00t/data/stats.py
Model stack
Backbone, action head, and reusable model plumbing.
- Model pipeline · Shared model pipeline abstraction and reusable wiring.
gr00t/model/base/model_pipeline.py - Qwen backbone · Vision-language processing backbone used by GR00T N1.7.
gr00t/model/modules/qwen3_backbone.py - Action head · Continuous action generation head for policy outputs.
gr00t/model/modules/flowmatching_modules.py - GR00T N1.7 · Top-level model combining perception, language, and action generation.
gr00t/model/gr00t_n1d7/gr00t_n1d7.py
Training plane
Config-driven experiment orchestration and distributed fine-tuning.
- Training config · Model, data, and training settings for launches and finetuning.
gr00t/configs/training/training_config.py - Trainer · Distributed training and fine-tuning execution logic.
gr00t/experiment/trainer.py - Launchers · Main training and finetuning entry points.
gr00t/experiment/launch_train.py
Runtime plane
Policy execution, serving, and evaluation paths.
- Gr00tPolicy · Policy abstraction used for inference, replay, and deployment.
gr00t/policy/gr00t_policy.py - ZMQ server · Serves policy actions over ZMQ for remote clients.
gr00t/eval/run_gr00t_server.py - Simulation eval · Simulation adapters and rollout wrappers for benchmark execution.
gr00t/eval/sim - Real robot eval · Real-robot evaluation entry points and robot-specific control loops.
gr00t/eval/real_robot/SO100/eval_so100.py - Open-loop eval · Compares predicted actions against recorded trajectories and visualizes results.
gr00t/eval/open_loop_eval.py
Deployment toolchain
Export and acceleration tooling for inference runtimes.
- Deployment tools · ONNX export, TensorRT engine building, and TRT wiring scripts.
scripts/deployment
Benchmarks
Example workflows and external benchmark integrations.
- Examples · Reproducible benchmark workflows and modality configs.
examples - External deps · Vendored benchmark dependencies treated as integration inputs.
external_dependencies
Connections 23
- Robot demos → LeRobot dataset: convert · Raw demonstrations are converted into normalized dataset shards.
- LeRobot dataset → Dataset loader: load · Dataset loader reads the LeRobot-style structure.
- Dataset loader → State actions: normalize · Samples are transformed into model-ready state and action tensors.
- Embodiment tags → Dataset loader: select · Embodiment tags steer state, action, and video field selection.
- Embodiment configs → Embodiment tags: define · Configs define the embodiment-specific modality bridge.
- Stats & video → Dataset loader: support · Statistics and video utilities support dataset ingestion.
- State actions → Collators: batch · Processed actions and states are assembled into training batches.
- Collators → Training config: feed · Configures how data is batched for training and finetuning.
- Training config → Trainer: drive · Training configuration drives distributed execution.
- Trainer → Launchers: run · Launchers invoke the trainer for train or finetune workflows.
- Collators → GR00T N1.7: input · Batched observations and actions are consumed by the model.
- Model pipeline → GR00T N1.7: compose · Shared pipeline plumbing composes the top-level model.
- Qwen backbone → GR00T N1.7: encode · Vision-language features feed the policy model.
- Action head → GR00T N1.7: predict · Action head produces continuous control outputs.
- GR00T N1.7 → Gr00tPolicy: wrap · Trained model is exposed through the runtime policy interface.
- Gr00tPolicy → ZMQ server: serve · Policy outputs are served over ZMQ to remote clients.
- Gr00tPolicy → Open-loop eval: evaluate · Offline evaluation reuses the same policy contract.
- ZMQ server → Simulation eval: control · Simulation clients request actions from the policy server.
- ZMQ server → Real robot eval: control · Real-robot controllers can consume remote policy actions.
- Deployment tools → ZMQ server: accelerate · Export and TensorRT tooling prepare faster serving paths.
- Examples → LeRobot dataset: adapt · Benchmarks provide modality configs and dataset conversion helpers.
- Examples → Gr00tPolicy: invoke · Example workflows exercise the shared policy interface.
- External deps → Examples: support · External benchmark repositories back the example integrations.