How d-robotics/lerobot works
Project Type and Purpose
• LeRobot is a modular Python library and CLI toolkit for real‑world robotics research, built on PyTorch and Hugging Face ecosystems.
• It bundles:
– Core API (“lerobot” package) for datasets, environments, policies, and robot‑device drivers
– CLI scripts for training, evaluation, data conversion, and robot control
– Example notebooks, benchmarks, and Docker configurations for CPU/GPU
High‑Level Components
A. Core Library (“lerobot” package)
1. configs
• Dataclass configurations for CLI scripts, policies, training, evaluation, environments
2. common
• datasets: LeRobotDataset format, serialization, Hugging Face Datasets integration
• envs: wrappers and factory for Gymnasium‑based simulation (aloha, pusht, xarm)
• policies: plugin implementations (ACT, Diffusion, TDMPC, VQ‑BeT, etc.) + factory loader
• robot_devices: hardware abstractions for cameras and motors (Dynamixel, Feetech, OpenCV)
• utils: logging, I/O, random, benchmark, wandb integration, hub upload
3. scripts
• CLI entrypoints: train.py, eval.py, control_robot.py, push_dataset_to_hub.py, visualize_dataset.py
4. templates
• HTML templates for dataset visualization
B. Examples & Benchmarks
• examples/: step‑by‑step usage tutorials (load data, train, evaluate, real robot control)
• benchmarks/: video and performance measurement scripts
C. Packaging & DevOps
• Docker/: CPU, GPU, dev images
• GitHub Actions workflows for CI (build, test, lint, nightly tests, trufflehog)
• pyproject.toml, Makefile, pre‑commit config for local builds and formatting
D. Tests
• pytest suites covering datasets, envs, policies, robot_devices mock tests, utils
External Dependencies & Services
• PyTorch (models, training loops)
• Hugging Face Datasets & Hub (remote dataset/model storage)
• Gymnasium Environments (aloha, xarm, pusht)
• FFmpeg / PyAV for video serialization
• Weights & Biases for experiment logging
• Docker for containerized builds
Component Relationships & Data Flow
– Configs → common.datasets to load episodes from HF Hub or local storage.
– common.datasets feeds batched data into common.policies which uses PyTorch models.
– Training/Evaluation loops (in train.py, eval.py) orchestrate data → model → optimizer → metrics → wandb.
– common.envs factory creates simulation environments consumed by policies at runtime.
– control_robot.py uses common.robot_devices to talk to real hardware (motors, cameras), wrapping data into same format as simulation.
Key Architectural Patterns
• Factory Pattern: for datasets, environments, policies → allows plug‑and‑play extensions
• Layered CLI Architecture: separation between “presentation” (CLI), “application logic” (scripts + configs), and “domain modules” (common)
• Plugin/Registry: each policy/dataset registers itself in a factory for dynamic loading
• Modular packaging: each concern in its own subpackage for isolation and testability