# nvidia/isaac-gr00t architecture

> Architecture diagram of the GitHub repository nvidia/isaac-gr00t, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvidia/isaac-gr00t
- Repository: https://github.com/nvidia/isaac-gr00t

## Overview

1. **What this repository is**
   - This is the reference implementation for NVIDIA Isaac GR00T N1.7, a vision-language-action robot policy stack.
   - The core runtime is a Python package (`gr00t`) plus scripts, examples, and benchmark-specific adapters for training, evaluation, and deployment.

2. **Primary system boundary**
   - The repo splits cleanly into two planes:
     - **Offline plane:** data preparation, dataset loading, training/fine-tuning, checkpoint export.
     - **Online plane:** policy serving, open-loop evaluation, closed-loop sim/real-robot control.
   - Most architecture diagrams should show data flowing from robot demos into a normalized dataset, then into model training, then into a policy server or inference script.

3. **Data layer**
   - The data format is a LeRobot v2-style dataset with GR00T-specific `meta/modality.json`.
   - Each dataset contains:
     - `meta/` for episode index, tasks, stats, and modality mapping
     - `data/` parquet files for state/action sequences
     - `videos/` mp4 recordings per observation stream
     - optional `masks/` for mask-guided experiments
   - This layer is where embodiment-specific state/action layouts are defined and interpreted.

4. **Dataset ingestion and normalization**
   - `gr00t/data/` handles dataset loading, sharded dataset variants, collators, statistics, and embodiment tags.
   - `state_action/` contains the logic for action representation, including relative end-effector processing and chunking.
   - The key architectural boundary here is between raw demonstration files and the model-ready tensors produced by the dataset/collator pipeline.

5. **Embodiment abstraction**
   - `gr00t/data/embodiment_tags.py` and `gr00t/configs/data/embodiment_configs.py` define the bridge between a robot/environment and the model’s expected modalities.
   - Embodiment tags select the state keys, action semantics, normalization, and video keys used at inference and training time.
   - This is the main mechanism that allows one model family to support DROID, LIBERO, SimplerEnv, and custom robots like SO100.

6. **Model architecture**
   - The model code lives under `gr00t/model/`.
   - The top-level GR00T N1.7 model combines:
     - a vision-language backbone (`qwen3_backbone.py` / GR00T N1.7 processing)
     - a diffusion/flow-matching action head for continuous action generation
   - `model/base/` and `model/modules/` separate reusable model pipeline pieces from architecture-specific components.

7. **Training and fine-tuning pipeline**
   - `gr00t/experiment/` is the training orchestration layer.
   - `launch_train.py` and `launch_finetune.py` are the main entry points; `trainer.py` encapsulates distributed training behavior.
   - Configs under `gr00t/configs/` define model, data, training, and fine-tuning settings.
   - The training path is designed for single-GPU and multi-GPU launches, including DeepSpeed configuration for scale-out fine-tuning.

8. **Policy and inference boundary**
   - `gr00t/policy/` turns a trained model into a runtime policy interface.
   - `Gr00tPolicy` is the main abstraction used by both standalone inference and the server-client deployment path.
   - `replay_policy.py` supports debugging by replaying dataset actions instead of using a learned model.

9. **Serving architecture for deployment**
   - `gr00t/eval/run_gr00t_server.py` exposes the policy over ZMQ as a lightweight action-serving process.
   - Clients send observations and receive actions, which keeps the heavy model isolated on a GPU host.
   - This separation is important for real robots and distributed evaluation: the control loop can stay near the robot while inference runs elsewhere.

10. **Evaluation stack**
    - `gr00t/eval/open_loop_eval.py` compares model-predicted actions against recorded trajectories and generates visual diagnostics.
    - `gr00t/eval/sim/` contains simulation adapters, wrappers, and environment utilities for benchmark execution.
    - `gr00t/eval/real_robot/` contains real-robot evaluation entry points, such as SO100-specific evaluation code.
    - The evaluation system is built around the same policy interface, so simulation and hardware can share the same inference contract.

11. **Benchmark and example integrations**
    - `examples/` contains reproducible benchmark workflows for DROID, LIBERO, SimplerEnv, SO100, and mask-guided background suppression.
    - These folders typically define benchmark-specific modality configs, dataset conversion helpers, and runnable evaluation/fine-tuning commands.
    - `external_dependencies/` holds benchmark repos that are treated as integration dependencies rather than core code.

12. **Deployment and acceleration toolchain**
    - `scripts/deployment/` contains export and acceleration paths:
      - ONNX export
      - TensorRT engine building
      - TRT pipeline wiring
      - platform-specific setup for dGPU, Orin, Thor, and Spark
    - This is the inference-optimization boundary: model code remains in Python, while deployment scripts bridge to platform-specific runtimes and GPU acceleration stacks.

13. **Video and media handling**
    - `gr00t/utils/video_utils.py` and related utilities support dataset video decoding and processing.
    - The repo explicitly supports different video backends depending on platform, with `torchcodec` emphasized for aarch64 systems.
    - This matters architecturally because visual observations are a first-class input to the policy, not an auxiliary asset.

14. **Tooling, packaging, and CI**
    - The repo uses `uv` for environment and dependency management, with platform-specific Dockerfiles under `docker/` and CI in `.github/workflows`.
    - Tests cover datasets, model forward paths, policy APIs, evaluation utilities, deployment scripts, and example workflows.
    - The test and CI structure mirrors the runtime architecture, which makes it easier to validate each boundary independently.

15. **How to draw the main data flow**
    - A good diagram would show:
      - raw robot/video demonstrations → dataset conversion/normalization → sharded LeRobot-style datasets
      - datasets + modality config + embodiment tag → model training/fine-tuning
      - trained checkpoint → standalone inference or ZMQ policy server
      - policy server → simulation client or real robot controller
      - evaluation outputs → metrics/plots/benchmark results
    - The central concept is that **embodiment tags and modality configs connect data, model, and deployment into one consistent control interface**.

Main entry points: [`gr00t/model/gr00t_n1d7/gr00t_n1d7.py`](https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/model/gr00t_n1d7/gr00t_n1d7.py) (GR00T N1.7), [`gr00t/policy/gr00t_policy.py`](https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/policy/gr00t_policy.py) (Gr00tPolicy), [`gr00t/eval/run_gr00t_server.py`](https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/eval/run_gr00t_server.py) (ZMQ server), [`gr00t/data/dataset/factory.py`](https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/data/dataset/factory.py) (Dataset loader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data plane"]
  node_raw_demos[("Robot demos<br/>source data")]
  node_dataset_layout[("LeRobot dataset<br/>dataset format")]
  node_dataset_loader["Dataset loader<br/>data ingestion<br/>[factory.py]"]
  node_state_action["State actions<br/>action processing"]
  node_embodiment_tags["Embodiment tags<br/>modality bridge<br/>[embodiment_tags.py]"]
  node_embodiment_cfg["Embodiment configs<br/>config mapping"]
  node_collators["Collators<br/>batching<br/>[collators.py]"]
  node_stats_video["Stats & video<br/>preprocess utils<br/>[stats.py]"]
end

subgraph group_model["Model stack"]
  node_model_pipeline["Model pipeline<br/>model plumbing<br/>[model_pipeline.py]"]
  node_qwen_backbone["Qwen backbone<br/>vision-language backbone<br/>[qwen3_backbone.py]"]
  node_action_head["Action head<br/>flow matching head"]
  node_gr00t_n1d7["GR00T N1.7<br/>policy model<br/>[gr00t_n1d7.py]"]
end

subgraph group_train["Training plane"]
  node_training_cfg["Training config<br/>experiment config<br/>[training_config.py]"]
  node_trainer["Trainer<br/>distributed training<br/>[trainer.py]"]
  node_launchers["Launchers<br/>entry points<br/>[launch_train.py]"]
end

subgraph group_runtime["Runtime plane"]
  node_gr00t_policy["Gr00tPolicy<br/>runtime policy<br/>[gr00t_policy.py]"]
  node_policy_server["ZMQ server<br/>policy serving"]
  node_sim_eval["Simulation eval<br/>sim benchmark"]
  node_real_eval["Real robot eval<br/>hardware benchmark<br/>[eval_so100.py]"]
  node_open_loop_eval["Open-loop eval<br/>offline eval<br/>[open_loop_eval.py]"]
end

subgraph group_deploy["Deployment toolchain"]
  node_deployment_tools["Deployment tools<br/>export/acceleration"]
end

subgraph group_integrations["Benchmarks"]
  node_examples["Examples<br/>benchmarks"]
  node_external_deps["External deps<br/>benchmark repos"]
end

node_raw_demos -->|"convert"| node_dataset_layout
node_dataset_layout -->|"load"| node_dataset_loader
node_dataset_loader -->|"normalize"| node_state_action
node_embodiment_tags -->|"select"| node_dataset_loader
node_embodiment_cfg -->|"define"| node_embodiment_tags
node_stats_video -->|"support"| node_dataset_loader
node_state_action -->|"batch"| node_collators
node_collators -->|"feed"| node_training_cfg
node_training_cfg -->|"drive"| node_trainer
node_trainer -->|"run"| node_launchers
node_collators -->|"input"| node_gr00t_n1d7
node_model_pipeline -->|"compose"| node_gr00t_n1d7
node_qwen_backbone -->|"encode"| node_gr00t_n1d7
node_action_head -->|"predict"| node_gr00t_n1d7
node_gr00t_n1d7 -->|"wrap"| node_gr00t_policy
node_gr00t_policy -->|"serve"| node_policy_server
node_gr00t_policy -->|"evaluate"| node_open_loop_eval
node_policy_server -->|"control"| node_sim_eval
node_policy_server -->|"control"| node_real_eval
node_deployment_tools -->|"accelerate"| node_policy_server
node_examples -->|"adapt"| node_dataset_layout
node_examples -->|"invoke"| node_gr00t_policy
node_external_deps -->|"support"| node_examples

click node_raw_demos "https://github.com/nvidia/isaac-gr00t/tree/main/demo_data"
click node_dataset_layout "https://github.com/nvidia/isaac-gr00t/tree/main/demo_data/cube_to_bowl_5"
click node_dataset_loader "https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/data/dataset/factory.py"
click node_state_action "https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/data/state_action/state_action_processor.py"
click node_embodiment_tags "https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/data/embodiment_tags.py"
click node_embodiment_cfg "https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/configs/data/embodiment_configs.py"
click node_collators "https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/data/collator/collators.py"
click node_stats_video "https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/data/stats.py"
click node_model_pipeline "https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/model/base/model_pipeline.py"
click node_qwen_backbone "https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/model/modules/qwen3_backbone.py"
click node_action_head "https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/model/modules/flowmatching_modules.py"
click node_gr00t_n1d7 "https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/model/gr00t_n1d7/gr00t_n1d7.py"
click node_training_cfg "https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/configs/training/training_config.py"
click node_trainer "https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/experiment/trainer.py"
click node_launchers "https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/experiment/launch_train.py"
click node_gr00t_policy "https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/policy/gr00t_policy.py"
click node_policy_server "https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/eval/run_gr00t_server.py"
click node_sim_eval "https://github.com/nvidia/isaac-gr00t/tree/main/gr00t/eval/sim"
click node_real_eval "https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/eval/real_robot/SO100/eval_so100.py"
click node_open_loop_eval "https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/eval/open_loop_eval.py"
click node_deployment_tools "https://github.com/nvidia/isaac-gr00t/tree/main/scripts/deployment"
click node_examples "https://github.com/nvidia/isaac-gr00t/tree/main/examples"
click node_external_deps "https://github.com/nvidia/isaac-gr00t/tree/main/external_dependencies"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw_demos,node_dataset_layout,node_dataset_loader,node_state_action,node_embodiment_tags,node_embodiment_cfg,node_collators,node_stats_video toneBlue
class node_model_pipeline,node_qwen_backbone,node_action_head,node_gr00t_n1d7 toneAmber
class node_training_cfg,node_trainer,node_launchers toneMint
class node_gr00t_policy,node_policy_server,node_sim_eval,node_real_eval,node_open_loop_eval toneRose
class node_deployment_tools toneIndigo
class node_examples,node_external_deps toneTeal
```

## Components

### Data plane

Dataset ingestion, normalization, and embodiment mapping.

- [Robot demos](https://github.com/nvidia/isaac-gr00t/tree/main/demo_data): source data · Raw robot demonstrations and recorded videos before normalization. · `demo_data`
- [LeRobot dataset](https://github.com/nvidia/isaac-gr00t/tree/main/demo_data/cube_to_bowl_5): dataset format · GR00T-style episode, meta, video, and optional mask layout. · `demo_data/cube_to_bowl_5`
- [Dataset loader](https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/data/dataset/factory.py): data ingestion · Loads episodes, sharded datasets, and dataset variants. · `gr00t/data/dataset/factory.py`
- [State actions](https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/data/state_action/state_action_processor.py): action processing · Action representation, chunking, and pose/state conversion. · `gr00t/data/state_action/state_action_processor.py`
- [Embodiment tags](https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/data/embodiment_tags.py): modality bridge · Maps robot-specific state, action, and video keys into a shared interface. · `gr00t/data/embodiment_tags.py`
- [Embodiment configs](https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/configs/data/embodiment_configs.py): config mapping · Config layer that selects modality and normalization rules per robot family. · `gr00t/configs/data/embodiment_configs.py`
- [Collators](https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/data/collator/collators.py): batching · Builds model-ready batches and normalized tensors from dataset samples. · `gr00t/data/collator/collators.py`
- [Stats & video](https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/data/stats.py): preprocess utils · Dataset statistics and video decoding/processing utilities. · `gr00t/data/stats.py`

### Model stack

Backbone, action head, and reusable model plumbing.

- [Model pipeline](https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/model/base/model_pipeline.py): model plumbing · Shared model pipeline abstraction and reusable wiring. · `gr00t/model/base/model_pipeline.py`
- [Qwen backbone](https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/model/modules/qwen3_backbone.py): vision-language backbone · Vision-language processing backbone used by GR00T N1.7. · `gr00t/model/modules/qwen3_backbone.py`
- [Action head](https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/model/modules/flowmatching_modules.py): flow matching head · Continuous action generation head for policy outputs. · `gr00t/model/modules/flowmatching_modules.py`
- [GR00T N1.7](https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/model/gr00t_n1d7/gr00t_n1d7.py): policy model · Top-level model combining perception, language, and action generation. · `gr00t/model/gr00t_n1d7/gr00t_n1d7.py`

### Training plane

Config-driven experiment orchestration and distributed fine-tuning.

- [Training config](https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/configs/training/training_config.py): experiment config · Model, data, and training settings for launches and finetuning. · `gr00t/configs/training/training_config.py`
- [Trainer](https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/experiment/trainer.py): distributed training · Distributed training and fine-tuning execution logic. · `gr00t/experiment/trainer.py`
- [Launchers](https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/experiment/launch_train.py): entry points · Main training and finetuning entry points. · `gr00t/experiment/launch_train.py`

### Runtime plane

Policy execution, serving, and evaluation paths.

- [Gr00tPolicy](https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/policy/gr00t_policy.py): runtime policy · Policy abstraction used for inference, replay, and deployment. · `gr00t/policy/gr00t_policy.py`
- [ZMQ server](https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/eval/run_gr00t_server.py): policy serving · Serves policy actions over ZMQ for remote clients. · `gr00t/eval/run_gr00t_server.py`
- [Simulation eval](https://github.com/nvidia/isaac-gr00t/tree/main/gr00t/eval/sim): sim benchmark · Simulation adapters and rollout wrappers for benchmark execution. · `gr00t/eval/sim`
- [Real robot eval](https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/eval/real_robot/SO100/eval_so100.py): hardware benchmark · Real-robot evaluation entry points and robot-specific control loops. · `gr00t/eval/real_robot/SO100/eval_so100.py`
- [Open-loop eval](https://github.com/nvidia/isaac-gr00t/blob/main/gr00t/eval/open_loop_eval.py): offline eval · Compares predicted actions against recorded trajectories and visualizes results. · `gr00t/eval/open_loop_eval.py`

### Deployment toolchain

Export and acceleration tooling for inference runtimes.

- [Deployment tools](https://github.com/nvidia/isaac-gr00t/tree/main/scripts/deployment): export/acceleration · ONNX export, TensorRT engine building, and TRT wiring scripts. · `scripts/deployment`

### Benchmarks

Example workflows and external benchmark integrations.

- [Examples](https://github.com/nvidia/isaac-gr00t/tree/main/examples): benchmarks · Reproducible benchmark workflows and modality configs. · `examples`
- [External deps](https://github.com/nvidia/isaac-gr00t/tree/main/external_dependencies): benchmark repos · Vendored benchmark dependencies treated as integration inputs. · `external_dependencies`

## Connections

- Robot demos → LeRobot dataset: convert · Raw demonstrations are converted into normalized dataset shards.
- LeRobot dataset → Dataset loader: load · Dataset loader reads the LeRobot-style structure.
- Dataset loader → State actions: normalize · Samples are transformed into model-ready state and action tensors.
- Embodiment tags → Dataset loader: select · Embodiment tags steer state, action, and video field selection.
- Embodiment configs → Embodiment tags: define · Configs define the embodiment-specific modality bridge.
- Stats & video → Dataset loader: support · Statistics and video utilities support dataset ingestion.
- State actions → Collators: batch · Processed actions and states are assembled into training batches.
- Collators → Training config: feed · Configures how data is batched for training and finetuning.
- Training config → Trainer: drive · Training configuration drives distributed execution.
- Trainer → Launchers: run · Launchers invoke the trainer for train or finetune workflows.
- Collators → GR00T N1.7: input · Batched observations and actions are consumed by the model.
- Model pipeline → GR00T N1.7: compose · Shared pipeline plumbing composes the top-level model.
- Qwen backbone → GR00T N1.7: encode · Vision-language features feed the policy model.
- Action head → GR00T N1.7: predict · Action head produces continuous control outputs.
- GR00T N1.7 → Gr00tPolicy: wrap · Trained model is exposed through the runtime policy interface.
- Gr00tPolicy → ZMQ server: serve · Policy outputs are served over ZMQ to remote clients.
- Gr00tPolicy → Open-loop eval: evaluate · Offline evaluation reuses the same policy contract.
- ZMQ server → Simulation eval: control · Simulation clients request actions from the policy server.
- ZMQ server → Real robot eval: control · Real-robot controllers can consume remote policy actions.
- Deployment tools → ZMQ server: accelerate · Export and TensorRT tooling prepare faster serving paths.
- Examples → LeRobot dataset: adapt · Benchmarks provide modality configs and dataset conversion helpers.
- Examples → Gr00tPolicy: invoke · Example workflows exercise the shared policy interface.
- External deps → Examples: support · External benchmark repositories back the example integrations.
