How facebookresearch/digit360 works
Main Components
• Hardware & Mechanical Design:
• Physical components such as the printed circuit board (PCB) stack (hardware design files in design/hardware), the custom hyperfisheye lens (design/lens), and the compliant, artificial fingertip (design/fingertip).
• Mechanical design elements (design/mechanical) that house or support these electronic and optical components.
• Firmware:
• Firmwares for different parts, such as the MCU, ICS controllers, and USB interface (firmware folders for ICS, MCU, and USB).
• Software Interface:
• Standalone Python API: Show the digit360/interface module which provides a python interface to access sensor modalities (like audio, image, LED, and USB examples under digit360/interface/examples).
• Udev and Setup Scripts: Include components like udev rules (digit360/udev) that help in device recognition, and setup scripts (install_digit360_venv.sh, setup.py) for environment setup.
• Data Processing & Training:
• Train directory which contains multimodal classification code and a training pipeline (train/multimodal). This shows how sensor data is used for machine learning and inference.
• Simulation & Visualization:
• From the hardware to firmware – show how sensor data from the physical modules is captured.
• From firmware to both the standalone Python interface and ROS2 middleware – indicate how the sensor’s multi-modal data is processed and published.
• For ROS2, show how sensor data flows into ROS nodes (publishers/subscribers) and is packaged using custom messages (d360_msgs).
• Within the training module, link the interface with data processing pipelines and the multimodal classifiers.
• Configuration and dependency relationships:
• Configuration files and setup tools should be noted as supporting the software components.
• Udev rules facilitate hardware communication with the OS.
Architectural Patterns & Design Principles
• Modularity: Emphasize how the project is divided into hardware (design, firmware), middleware (software interfaces), and application layers (training and simulation).
• Separation of Concerns: Illustrate clear separation between standalone package functionality versus ROS2 integration.
• Extensibility: Note that there are examples and proto message definitions to facilitate further development of sensor data handling.
• Integration Focus: Clearly show the integration points between hardware design documents, firmware images, software libraries, and simulation environments.