How spirit-ai-team/lerobot works
Project Type and Purpose
• LeRobot is a robotics framework/library designed for training, evaluating, and deploying policies on both simulation environments and real robots.
• It is a full-stack solution in the robotics domain, integrating datasets, pretrained policies, simulation environments (using Gymnasium), and live control of actual robot devices.
• The README highlights examples of running simulations (e.g., ALOHA, TDMPC, Diffusion Policy) and real-world applications (e.g., SO-100, LeKiwi).
File Structure for Architectural Components
• Top-level directories like “lerobot”, “examples”, “benchmarks”, “docker”, “tests” indicate modular organization.
• The “lerobot” directory is the core package:
• Contains subdirectories such as “configs” (for configuration management), “common” (for datasets, environments, policies, robot devices, and utilities), “scripts” (executable command-line tools for training, evaluation, visualization, etc.) and “templates” (for HTML visualizations).
• The “benchmarks” and “examples” directories provide use-case scripts and demonstrations.
• Docker-related directories detail containerization for CPU and GPU variants, indicating deployment aspects.
• The tests directory ensures continuous integration and unit testing of individual modules.
Main Components
a. Core Library (lerobot package)
• Configurations: “lerobot/configs” managing default, training, evaluation, and policy-specific options.
• Datasets and Data Handling: “lerobot/common/datasets” that organize human demonstration datasets, including support for multiple dataset formats (e.g., hdf5, safetensors, zarr).
• Simulation Environments: “lerobot/common/envs” which sets up environments for Gymnasium-based simulation.
• Policies: “lerobot/common/policies” with submodules for different policy types (ACT, Diffusion, TDMPC, VQ-BeT, etc.) that encapsulate model configuration and inference logic.
• Robot Devices: “lerobot/common/robot_devices” to interface with real hardware (motors, cameras, etc.).
• Utilities: “lerobot/common/utils” providing support functions for logging, data I/O, randomness, and benchmark utilities.
b. Executable Scripts
• Training Script (lerobot/scripts/train.py)
• Evaluation Script (lerobot/scripts/eval.py)
• Control Script for real robots (lerobot/scripts/control_robot.py)
• Data Visualization and Upload Tools (visualize_dataset.py, push_dataset_to_hub.py)
c. External Dependencies & Services
• Integration with Hugging Face Hub for datasets and pretrained policies.
• WandB for logging and experiment tracking.
• Gymnasium environments and Docker for simulated/real-world execution.
d. Docker & Deployment
• The “docker” directory indicates containerization support for different configurations (e.g., GPU, CPU, GPU dev).
e. Examples and Benchmarks
• Serve as “client” examples showing how to use the LeRobot API to load datasets, train policies, and control robots.
Relationships and Data Flow
• Data/Configuration Ingestion: Users start by loading configurations (via “lerobot/configs”) or datasets (using LeRobotDataset from “lerobot/common/datasets”) from local or Hugging Face hub sources.
• Training/Evaluation Flow: The training script (train.py) interacts with policies, datasets, and environment modules while using the configuration modules to set hyperparameters. Evaluation script (eval.py) similarly calls policies and environment modules.
• Real-World Execution: The control scripts take commands from a user and interface with robot devices (hardware interfacing via “lerobot/common/robot_devices”).
• Experiment Tracking: Integration with WandB is used during training/evaluation, represented as an external logging service.
• Docker Deployment and CI: The Docker and GitHub workflows (inside .github/workflows) ensure reproducible builds and tests.
Architectural Patterns and Design Principles
• Modular Separation of Concerns: Each module (configs, datasets, policies, environments, devices, etc.) is distinctly separated, making the architecture easy to extend or swap components.
• Pipeline-based Flow: Data is ingested, processed through simulation or real-robot control pathways, and then results (checkpoints, logs, videos) are produced.
• Flexibility and Extensibility: The design supports multiple types of policies and datasets and enables easy addition of new simulations or real device interfaces.
• Clear Dependency Injection: Scripts use configurations to abstract hardware details and policy parameters.