nvlabs/industreallib

1. What this repository is

IndustRealLib is a Python deployment library for running a subset of IndustReal’s real-robot assembly policies on a Franka Panda. It is not a training monolith; the runtime focus is on taking perception outputs and RL policy checkpoints, then driving the robot through Franka-facing control.

2. Top-level execution boundary

The main runtime starts from two CLI entry points in scripts/: run_task.py for a single task instance and run_sequence.py for a multi-task sequence instance. run_simple_action.py is a small control-test harness for direct robot actions such as homing, gripper open/close, and pose moves.

3. External systems and hardware

The architecture depends on a Franka robot controlled through frankapy and franka-interface, plus an Intel RealSense D400 camera mounted on the robot. Isaac Gym is used offline to train the shipped RL policies, but at runtime the repo consumes pretrained checkpoints rather than simulating. AprilTags are part of the perception/calibration loop.

4. Package-level subsystem split

Under src/industreallib/, the code is divided into four main runtime subsystems: control, perception, rl, and tasks, with sequences layered on top of tasks. This is the key architectural boundary: perception finds targets, RL decides motion, control sends robot commands, and task/sequence classes glue those pieces together.

5. Control subsystem

control/scripts/control_utils.py contains thin wrappers around Franka motion and gripper operations. These are the low-level actions used by run_simple_action.py and by task classes before/after a policy-driven motion, such as going home, moving upward, or opening/closing the gripper.

6. Perception subsystem

perception/ contains the camera-to-world calibration and object localization pipeline. The runtime flow starts with extrinsics from JSON, then workspace mapping using an AprilTag reference, then object detection using checkpoints such as industreal_detector_pegs.pt, ..._gears.pt, and ..._connectors.pt. The outputs are stored as object detections and workspace mappings in the local io/ files.

7. Perception configuration and calibration data

perception/configs/perception.yaml selects scene-specific settings, checkpoint choice, and mapping behavior. The perception/scripts/ tools are operational utilities rather than always-on services: calibrate_extrinsics.py calibrates camera pose, map_workspace.py builds the image-to-world mapping, and detect_objects.py performs inference. The repo ships the calibration and mapping artifacts as JSON files under perception/io/.

8. RL policy subsystem

rl/checkpoints/ stores pretrained policies, each as a checkpoint directory with a config.yaml and a .pth network file. These policies are organized by skill, such as reach, pick, place, and insert, plus a generic policy. At runtime, task classes load these checkpoints and generate actions from the current observation vector.

9. Task subsystem

tasks/classes/ defines the core execution model. IndustRealTaskBase provides common behavior: load policy, get observations, turn policy actions into pose targets, and execute pre/post-goal procedures. The concrete subclasses IndustRealTaskReach, ...Pick, ...Place, and ...Insert specialize only the task-specific observation logic and task behavior.

10. Task instance configurations

tasks/instance_configs/ contains YAML descriptions of concrete task setups, such as pick_pegs.yaml, insert_gears.yaml, and reach.yaml. These configs bind a generic task class to a specific object type, policy choice, control parameters, and scene assumptions. They are the primary mechanism for varying behavior without changing code.

11. Sequence subsystem

sequences/classes/industreal_sequence_base.py provides orchestration for chaining tasks in a fixed order. Sequence instance configs in sequences/instance_configs/ specify which task instances to run and which objects each task should act on. In effect, sequences are a higher-level scheduler over task instances, not a separate control stack.

12. Data flow from camera to robot

The typical runtime path is: camera and calibration data are read, AprilTag-based workspace mapping converts image coordinates to robot-frame goals, object detection identifies target parts, the task instance converts those goals into task observations, the RL policy produces actions, and control utilities send the resulting pose/gripper commands to Franka. This flow is repeated once per object or per stage in a sequence.

13. Operational artifacts and persistence

The repo includes shipped artifacts needed for deployment: detector checkpoints, RL policy checkpoints, extrinsics calibration files, workspace mapping files, and task/sequence YAMLs. These assets define the runtime environment as much as the code does, so an architecture diagram should show them as configuration and model inputs feeding the live pipeline.

14. Training code is auxiliary

The perception/scripts/training/ directory contains Mask R-CNN-style training utilities for object detection. This code is useful for reproducing or updating perception models, but it is not in the main deployment loop. The live system consumes the trained checkpoint artifacts produced from this area.

15. Main architectural boundary to draw

The cleanest diagram boundary is between the physical robot/camera world and the Python deployment stack. Inside the stack, show three layers: perception produces goals, tasks/policies produce skill-level motion, and control executes robot commands. Above them, run_task.py and run_sequence.py select instances through YAML configs, while run_simple_action.py bypasses perception and RL for direct robot tests.