# nvlabs/industreallib architecture

> Architecture diagram of the GitHub repository nvlabs/industreallib, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvlabs/industreallib
- Repository: https://github.com/nvlabs/industreallib

## Overview

1. **What this repository is**
   IndustRealLib is a Python deployment library for running a subset of IndustReal’s real-robot assembly policies on a Franka Panda. It is not a training monolith; the runtime focus is on taking perception outputs and RL policy checkpoints, then driving the robot through Franka-facing control.

2. **Top-level execution boundary**
   The main runtime starts from two CLI entry points in `scripts/`: `run_task.py` for a single task instance and `run_sequence.py` for a multi-task sequence instance. `run_simple_action.py` is a small control-test harness for direct robot actions such as homing, gripper open/close, and pose moves.

3. **External systems and hardware**
   The architecture depends on a Franka robot controlled through `frankapy` and `franka-interface`, plus an Intel RealSense D400 camera mounted on the robot. Isaac Gym is used offline to train the shipped RL policies, but at runtime the repo consumes pretrained checkpoints rather than simulating. AprilTags are part of the perception/calibration loop.

4. **Package-level subsystem split**
   Under `src/industreallib/`, the code is divided into four main runtime subsystems: `control`, `perception`, `rl`, and `tasks`, with `sequences` layered on top of tasks. This is the key architectural boundary: perception finds targets, RL decides motion, control sends robot commands, and task/sequence classes glue those pieces together.

5. **Control subsystem**
   `control/scripts/control_utils.py` contains thin wrappers around Franka motion and gripper operations. These are the low-level actions used by `run_simple_action.py` and by task classes before/after a policy-driven motion, such as going home, moving upward, or opening/closing the gripper.

6. **Perception subsystem**
   `perception/` contains the camera-to-world calibration and object localization pipeline. The runtime flow starts with extrinsics from JSON, then workspace mapping using an AprilTag reference, then object detection using checkpoints such as `industreal_detector_pegs.pt`, `..._gears.pt`, and `..._connectors.pt`. The outputs are stored as object detections and workspace mappings in the local `io/` files.

7. **Perception configuration and calibration data**
   `perception/configs/perception.yaml` selects scene-specific settings, checkpoint choice, and mapping behavior. The `perception/scripts/` tools are operational utilities rather than always-on services: `calibrate_extrinsics.py` calibrates camera pose, `map_workspace.py` builds the image-to-world mapping, and `detect_objects.py` performs inference. The repo ships the calibration and mapping artifacts as JSON files under `perception/io/`.

8. **RL policy subsystem**
   `rl/checkpoints/` stores pretrained policies, each as a checkpoint directory with a `config.yaml` and a `.pth` network file. These policies are organized by skill, such as `reach`, `pick`, `place`, and `insert`, plus a generic policy. At runtime, task classes load these checkpoints and generate actions from the current observation vector.

9. **Task subsystem**
   `tasks/classes/` defines the core execution model. `IndustRealTaskBase` provides common behavior: load policy, get observations, turn policy actions into pose targets, and execute pre/post-goal procedures. The concrete subclasses `IndustRealTaskReach`, `...Pick`, `...Place`, and `...Insert` specialize only the task-specific observation logic and task behavior.

10. **Task instance configurations**
   `tasks/instance_configs/` contains YAML descriptions of concrete task setups, such as `pick_pegs.yaml`, `insert_gears.yaml`, and `reach.yaml`. These configs bind a generic task class to a specific object type, policy choice, control parameters, and scene assumptions. They are the primary mechanism for varying behavior without changing code.

11. **Sequence subsystem**
   `sequences/classes/industreal_sequence_base.py` provides orchestration for chaining tasks in a fixed order. Sequence instance configs in `sequences/instance_configs/` specify which task instances to run and which objects each task should act on. In effect, sequences are a higher-level scheduler over task instances, not a separate control stack.

12. **Data flow from camera to robot**
   The typical runtime path is: camera and calibration data are read, AprilTag-based workspace mapping converts image coordinates to robot-frame goals, object detection identifies target parts, the task instance converts those goals into task observations, the RL policy produces actions, and control utilities send the resulting pose/gripper commands to Franka. This flow is repeated once per object or per stage in a sequence.

13. **Operational artifacts and persistence**
   The repo includes shipped artifacts needed for deployment: detector checkpoints, RL policy checkpoints, extrinsics calibration files, workspace mapping files, and task/sequence YAMLs. These assets define the runtime environment as much as the code does, so an architecture diagram should show them as configuration and model inputs feeding the live pipeline.

14. **Training code is auxiliary**
   The `perception/scripts/training/` directory contains Mask R-CNN-style training utilities for object detection. This code is useful for reproducing or updating perception models, but it is not in the main deployment loop. The live system consumes the trained checkpoint artifacts produced from this area.

15. **Main architectural boundary to draw**
   The cleanest diagram boundary is between the physical robot/camera world and the Python deployment stack. Inside the stack, show three layers: perception produces goals, tasks/policies produce skill-level motion, and control executes robot commands. Above them, `run_task.py` and `run_sequence.py` select instances through YAML configs, while `run_simple_action.py` bypasses perception and RL for direct robot tests.

Main entry points: [`src/industreallib/tasks/classes/industreal_task_base.py`](https://github.com/nvlabs/industreallib/blob/main/src/industreallib/tasks/classes/industreal_task_base.py) (task base), [`src/industreallib/perception/scripts/detect_objects.py`](https://github.com/nvlabs/industreallib/blob/main/src/industreallib/perception/scripts/detect_objects.py) (object detect), [`src/industreallib/perception/scripts/calibrate_extrinsics.py`](https://github.com/nvlabs/industreallib/blob/main/src/industreallib/perception/scripts/calibrate_extrinsics.py) (calibration), [`src/industreallib/control/scripts/control_utils.py`](https://github.com/nvlabs/industreallib/blob/main/src/industreallib/control/scripts/control_utils.py) (control utils)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime entrypoints"]
  node_scripts_run_task["run task<br/>cli entrypoint<br/>[run_task.py]"]
  node_scripts_run_sequence["run sequence<br/>cli entrypoint<br/>[run_sequence.py]"]
  node_scripts_run_simple_action["simple action<br/>cli harness"]
end

subgraph group_stack["Deployment stack"]
  node_control_utils["control utils<br/>Franka control wrappers<br/>[control_utils.py]"]
  node_perception_calibration["calibration<br/>camera pose & mapping"]
  node_perception_detection["object detect<br/>vision inference<br/>[detect_objects.py]"]
  node_task_base["task base<br/>task execution core"]
  node_task_variants["task variants<br/>task specializations"]
  node_sequence_base["sequence base<br/>task scheduler"]
  node_sequence_utils["sequence utils<br/>sequence helpers<br/>[sequence_utils.py]"]
end

subgraph group_assets["Operational assets"]
  node_perception_config["perception cfg<br/>scene config<br/>[perception.yaml]"]
  node_detector_checkpoints[("detectors<br/>vision checkpoints")]
  node_perception_io["perception I/O<br/>calibration data"]
  node_task_configs["task configs<br/>task instance YAMLs"]
  node_policy_checkpoints[("policy checkpoints<br/>RL policies")]
  node_sequence_configs["sequence configs<br/>sequence YAMLs"]
  node_training_tools["training tools<br/>auxiliary training"]
end

subgraph group_external["Physical systems"]
  node_franka_robot{{"Franka Panda<br/>physical robot"}}
  node_realsense_camera{{"RealSense D400<br/>depth camera"}}
  node_frankapy_iface{{"Franka libs<br/>robot middleware"}}
  node_apriltags{{"AprilTags<br/>fiducial reference"}}
end

node_scripts_run_task -->|"loads"| node_task_base
node_scripts_run_sequence -->|"loads"| node_sequence_base
node_scripts_run_simple_action -->|"calls"| node_control_utils
node_sequence_base -->|"runs"| node_task_base
node_sequence_base -->|"reads"| node_sequence_configs
node_sequence_utils -.->|"supports"| node_sequence_base
node_task_base -->|"specializes"| node_task_variants
node_task_base -->|"loads"| node_policy_checkpoints
node_task_base -->|"executes"| node_control_utils
node_task_base -->|"configured by"| node_task_configs
node_task_base -->|"consumes"| node_perception_detection
node_task_base -.->|"consumes"| node_perception_io
node_perception_detection -->|"uses"| node_detector_checkpoints
node_perception_detection -->|"reads"| node_perception_config
node_perception_calibration -->|"references"| node_apriltags
node_perception_calibration -->|"calibrates"| node_realsense_camera
node_perception_calibration -->|"writes"| node_perception_io
node_perception_detection -->|"writes"| node_perception_io
node_control_utils -->|"depends on"| node_frankapy_iface
node_control_utils -->|"drives"| node_franka_robot
node_scripts_run_task -.->|"invokes"| node_perception_detection
node_scripts_run_task -.->|"invokes"| node_perception_calibration
node_scripts_run_sequence -.->|"invokes"| node_perception_detection
node_training_tools -.->|"produces"| node_detector_checkpoints

click node_scripts_run_task "https://github.com/nvlabs/industreallib/blob/main/scripts/run_task.py"
click node_scripts_run_sequence "https://github.com/nvlabs/industreallib/blob/main/scripts/run_sequence.py"
click node_scripts_run_simple_action "https://github.com/nvlabs/industreallib/blob/main/scripts/run_simple_action.py"
click node_control_utils "https://github.com/nvlabs/industreallib/blob/main/src/industreallib/control/scripts/control_utils.py"
click node_perception_calibration "https://github.com/nvlabs/industreallib/blob/main/src/industreallib/perception/scripts/calibrate_extrinsics.py"
click node_perception_detection "https://github.com/nvlabs/industreallib/blob/main/src/industreallib/perception/scripts/detect_objects.py"
click node_perception_config "https://github.com/nvlabs/industreallib/blob/main/src/industreallib/perception/configs/perception.yaml"
click node_detector_checkpoints "https://github.com/nvlabs/industreallib/tree/main/src/industreallib/perception/checkpoints"
click node_perception_io "https://github.com/nvlabs/industreallib/tree/main/src/industreallib/perception/io"
click node_task_base "https://github.com/nvlabs/industreallib/blob/main/src/industreallib/tasks/classes/industreal_task_base.py"
click node_task_variants "https://github.com/nvlabs/industreallib/tree/main/src/industreallib/tasks/classes"
click node_task_configs "https://github.com/nvlabs/industreallib/tree/main/src/industreallib/tasks/instance_configs"
click node_policy_checkpoints "https://github.com/nvlabs/industreallib/tree/main/src/industreallib/rl/checkpoints"
click node_sequence_base "https://github.com/nvlabs/industreallib/blob/main/src/industreallib/sequences/classes/industreal_sequence_base.py"
click node_sequence_configs "https://github.com/nvlabs/industreallib/tree/main/src/industreallib/sequences/instance_configs"
click node_sequence_utils "https://github.com/nvlabs/industreallib/blob/main/src/industreallib/sequences/scripts/sequence_utils.py"
click node_training_tools "https://github.com/nvlabs/industreallib/tree/main/src/industreallib/perception/scripts/training"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_scripts_run_task,node_scripts_run_sequence,node_scripts_run_simple_action toneBlue
class node_control_utils,node_perception_calibration,node_perception_detection,node_task_base,node_task_variants,node_sequence_base,node_sequence_utils toneAmber
class node_perception_config,node_detector_checkpoints,node_perception_io,node_task_configs,node_policy_checkpoints,node_sequence_configs,node_training_tools toneMint
class node_franka_robot,node_realsense_camera,node_frankapy_iface,node_apriltags toneRose
```

## Components

### Runtime entrypoints

CLI entry points that start deployment or direct robot actions.

- [run task](https://github.com/nvlabs/industreallib/blob/main/scripts/run_task.py): cli entrypoint · Starts a single task instance from YAML configuration. · `scripts/run_task.py`
- [run sequence](https://github.com/nvlabs/industreallib/blob/main/scripts/run_sequence.py): cli entrypoint · Starts a multi-task sequence from sequence configuration. · `scripts/run_sequence.py`
- [simple action](https://github.com/nvlabs/industreallib/blob/main/scripts/run_simple_action.py): cli harness · Bypasses perception and RL for direct robot control tests. · `scripts/run_simple_action.py`

### Deployment stack

Core Python subsystems for perception, tasks, sequences, and control.

- [control utils](https://github.com/nvlabs/industreallib/blob/main/src/industreallib/control/scripts/control_utils.py): Franka control wrappers · Thin wrappers for homing, gripper, and pose commands. · `src/industreallib/control/scripts/control_utils.py`
- [calibration](https://github.com/nvlabs/industreallib/blob/main/src/industreallib/perception/scripts/calibrate_extrinsics.py): camera pose & mapping · Extrinsics calibration and workspace mapping around AprilTag reference data. · `src/industreallib/perception/scripts/calibrate_extrinsics.py`
- [object detect](https://github.com/nvlabs/industreallib/blob/main/src/industreallib/perception/scripts/detect_objects.py): vision inference · Runs detector inference to localize target parts in the scene. · `src/industreallib/perception/scripts/detect_objects.py`
- [task base](https://github.com/nvlabs/industreallib/blob/main/src/industreallib/tasks/classes/industreal_task_base.py): task execution core · Loads policies, builds observations, converts actions to robot targets, and handles task lifecycle. · `src/industreallib/tasks/classes/industreal_task_base.py`
- [task variants](https://github.com/nvlabs/industreallib/tree/main/src/industreallib/tasks/classes): task specializations · Reach, pick, place, and insert task classes that define task-specific observation and behavior. · `src/industreallib/tasks/classes`
- [sequence base](https://github.com/nvlabs/industreallib/blob/main/src/industreallib/sequences/classes/industreal_sequence_base.py): task scheduler · Orchestrates fixed-order execution of multiple task instances. · `src/industreallib/sequences/classes/industreal_sequence_base.py`
- [sequence utils](https://github.com/nvlabs/industreallib/blob/main/src/industreallib/sequences/scripts/sequence_utils.py): sequence helpers · Utilities supporting sequence setup and execution. · `src/industreallib/sequences/scripts/sequence_utils.py`

### Operational assets

Shipped checkpoints, calibration files, and instance configs used at runtime.

- [perception cfg](https://github.com/nvlabs/industreallib/blob/main/src/industreallib/perception/configs/perception.yaml): scene config · Selects perception scene behavior, checkpoints, and mapping options. · `src/industreallib/perception/configs/perception.yaml`
- [detectors](https://github.com/nvlabs/industreallib/tree/main/src/industreallib/perception/checkpoints): vision checkpoints · Pretrained object detectors for pegs, gears, and connectors. · `src/industreallib/perception/checkpoints`
- [perception I/O](https://github.com/nvlabs/industreallib/tree/main/src/industreallib/perception/io): calibration data · Saved extrinsics, workspace mapping, detections, and camera parameters. · `src/industreallib/perception/io`
- [task configs](https://github.com/nvlabs/industreallib/tree/main/src/industreallib/tasks/instance_configs): task instance YAMLs · Concrete task setups binding task class, object type, policy, and control parameters. · `src/industreallib/tasks/instance_configs`
- [policy checkpoints](https://github.com/nvlabs/industreallib/tree/main/src/industreallib/rl/checkpoints): RL policies · Pretrained skill policies with config and network weights for reach, pick, place, insert, and generic use. · `src/industreallib/rl/checkpoints`
- [sequence configs](https://github.com/nvlabs/industreallib/tree/main/src/industreallib/sequences/instance_configs): sequence YAMLs · Defines which task instances run and which objects each stage targets. · `src/industreallib/sequences/instance_configs`
- [training tools](https://github.com/nvlabs/industreallib/tree/main/src/industreallib/perception/scripts/training): auxiliary training · Offline perception training utilities used to produce detector checkpoints, not part of the live deployment loop. · `src/industreallib/perception/scripts/training`

### Physical systems

Robot, camera, and external libraries that the stack drives or depends on.

- **Franka Panda**: physical robot · Real robot arm and gripper driven by the deployment stack.
- **RealSense D400**: depth camera · Mounted camera providing runtime visual observations.
- **Franka libs**: robot middleware · External Franka control libraries used to send motions and gripper actions.
- **AprilTags**: fiducial reference · Reference markers used in calibration and workspace mapping.

## Connections

- run task → task base: loads · Task CLI instantiates a task from configuration.
- run sequence → sequence base: loads · Sequence CLI instantiates the sequence orchestrator.
- simple action → control utils: calls · Direct action harness invokes low-level control wrappers.
- sequence base → task base: runs · Sequences coordinate task instances in order.
- sequence base → sequence configs: reads · Sequence orchestration is driven by sequence YAMLs.
- sequence utils → sequence base: supports · Helpers assist sequence setup and execution.
- task base → task variants: specializes · Concrete task classes extend the task base.
- task base → policy checkpoints: loads · Tasks load pretrained policies for skill execution.
- task base → control utils: executes · Task actions are converted to robot commands through control utilities.
- task base → task configs: configured by · Task instances are selected and parameterized by YAML.
- task base → object detect: consumes · Task observations depend on detected object targets.
- task base → perception I/O: consumes · Tasks use calibration and mapping artifacts.
- object detect → detectors: uses · Inference runs on shipped detector weights.
- object detect → perception cfg: reads · Detection behavior is selected by perception configuration.
- calibration → AprilTags: references · Workspace mapping relies on AprilTag calibration reference.
- calibration → RealSense D400: calibrates · Calibration aligns camera pose to the robot/world frame.
- calibration → perception I/O: writes · Calibration and mapping artifacts are stored in local IO files.
- object detect → perception I/O: writes · Object detections are persisted for downstream task use.
- control utils → Franka libs: depends on · Control wrappers sit on top of Franka middleware.
- control utils → Franka Panda: drives · Commands are issued to the real robot and gripper.
- run task → object detect: invokes · Single-task execution starts the perception pipeline.
- run task → calibration: invokes · Task runs rely on calibrated workspace mapping.
- run sequence → object detect: invokes · Sequence stages consume perception outputs as needed.
- training tools → detectors: produces · Offline training utilities generate the detector artifacts.