How anhbantre/personfollowingrobot works
Project Type & Purpose
• The project is a robotics application with a strong focus on computer vision and AI. Its purpose is to create a smart trolley— a person-following robot— that can handle complex indoor environments by using 3D vision, deep learning-based object detection, and single object tracking.
• The README describes the use of a RealSense camera for depth sensing, ROS integration for middleware, and deep learning models (jetson-inference and Stark) converted to TensorRT for accelerated inference.
File Structure
• At the root, you have configuration files (.gitignore, LICENSE, and installation instructions for pyrealsense2), and two key Python files: human_detection.py and image_processing.py.
• The imgs directory holds various visual outputs and example images of floor detection and tracking, which suggests the importance of imaging in the pipeline.
• This indicates two primary software components: one dealing with human detection (likely interfacing with the deep learning model for tracking) and another focusing on image processing (preparing data for analysis, possibly including floor recognition).
Main Components
a. Hardware Components:
• Camera Sensor (Intel RealSense Depth D455): This is the primary sensor supplying both RGB and depth information.
• Board Computer (Jetson Nano): Processes the sensor data, runs the deep learning models, and handles ROS integration.
• Microcontroller (STM32F103C8T6): Likely used for low-level motor control and interfacing with the motor driver system.
• Motor Drivers & H-Bridge Driver: Interfaces that allow the control signals from either the Jetson Nano or microcontroller to drive the motors.
• Power System: Battery, voltage regulators, and associated circuitry.
b. Software Components:
• Sensor Driver Node: The ROS node or module that interfaces with the RealSense camera, capturing raw sensor data.
• Image Processing Node (image_processing.py): Processes the depth and image data to recognize floors and prepare data for detection.
• Human Detection/Tracking Node (human_detection.py): Uses deep learning models (using jetson-inference and Stark models, accelerated with TensorRT) to detect and track the person.
• ROS Communication Layer: The middleware that manages message passing between the various nodes, integrating sensor input with processing and control.
• Control/Actuation Node: Uses the output from detection & tracking to generate commands for the motors and drive the robot to follow the person.
Relationships and Interactions Between Components
• The RealSense camera sends raw data (both RGB and depth) to the Sensor Driver Node.
• This data is then passed to the Image Processing Node to perform floor recognition and potentially pre-filter the environment.
• Processed data is routed to the Human Detection/Tracking Node which employs deep learning techniques to locate and follow a person.
• All of these nodes communicate via ROS topics/services, ensuring real-time communication between the perception modules and the Control/Actuation Node.
• The Control node sends the appropriate drive commands (via ROS messages) to the microcontroller/motor drivers, which then actuate the robot’s motors.