How jugpanda/pickuppal works
1. Project shape
PickupPal is a robotics capstone repo centered on an autonomous “pickup” system. The repository is split into software for perception and decision-making, firmware for the embedded controller, and large image datasets/model artifacts for training and deployment.
2. Main runtime boundary: Python vs. Arduino
The clearest architectural boundary is between Python scripts under src/object-detection / src/yolo and the Arduino firmware under src/pickup-arduino. Python appears to run computer vision and higher-level autonomy, while the Arduino handles low-level actuation and sensor I/O for the robot.
3. Perception subsystem
Object detection is the core perception layer. It uses Ultralytics YOLO models and associated assets such as best.pt, yolo11n.pt, yolov8*.pt, and a TorchScript/NCNN export. The scripts yolo_detect.py, yolo.py, and pickup-learn.py suggest inference, experiment/training, and possibly robot interaction during learning or testing.
4. Autonomous control subsystem
src/object-detection/autonomous.py is the likely orchestration entry point for autonomous behavior. In architecture terms, it sits above perception and below operator interaction, turning detections into pickup/navigation decisions, likely by sending commands to the embedded controller.
5. Embedded control subsystem
src/pickup-arduino/autonomous.ino is the firmware boundary. It likely owns motors, servos, and sensors at real-time timescales. The Python layer should be treated as an upstream controller, with Arduino responsible for deterministic hardware actions and feedback.
6. Hardware validation and integration tests
src/tests/camera_test.py, motor_test.py, and sensor_test.py indicate separate hardware test paths rather than a single unified test harness. That implies the system was developed and validated component-by-component: camera input, motor output, and sensor readings.
7. Training data pipeline
There are two dataset areas: src/object-detection with data.yaml plus data_original.zip, and src/yolo/Lego with train/valid/test splits and labels. The presence of README.dataset.txt and README.roboflow.txt files suggests the datasets were prepared/imported via Roboflow and then organized for YOLO training.
8. Model lifecycle and conversion
The repo contains multiple model forms: raw PyTorch weights (.pt), a TorchScript export (yolo11n.torchscript), and an NCNN converted model directory (yolo11n_ncnn_model). That shows a deployment pipeline from training weights to optimized inference formats for edge or embedded-friendly execution.
9. Generated outputs and experiment artifacts
src/yolo/runs/detect/predict looks like a generated inference output folder rather than source code. It indicates the team ran local prediction experiments and saved results separately from the main code paths.
10. Likely end-to-end data flow
A reasonable architecture diagram would show: camera frames enter Python inference, YOLO produces detections, autonomous logic decides pickup behavior, and commands are sent to the Arduino controller, which drives motors and reads sensors. Sensor feedback then closes the loop back to the autonomy layer for continued operation.
11. What is intentionally not present
There is no sign of a web backend, database, or cloud service. This is primarily an offline/edge robotics system with local datasets, local model files, and hardware-attached control rather than a networked application.
12. Documentation and project context
The README is minimal, but the repo layout and notes point to a capstone build with split responsibilities: programming/circuitry, 3D modeling, and documentation. That matters architecturally because the software stack is only one part of a broader electromechanical system.
Main entry points: yolo_detect.py Detect · autonomous.ino Arduino · best.pt Best Weights · autonomous.py Autonomous
Components 22
Python stack
Perception, autonomy, model conversion, and local experiment scripts.
- Autonomous · High-level robot behavior that turns detections into pickup decisions.
src/object-detection/autonomous.py - Detect · Primary YOLO inference entry for camera-based object detection.
src/yolo/yolo_detect.py - YOLO Script · Supporting YOLO workflow script for running or testing detection logic.
src/object-detection/yolo.py - Learn · Robot learning or experimentation script tied to object detection workflows.
src/object-detection/pickup-learn.py - Convert · Converts trained YOLO models into an optimized NCNN deployment format.
src/object-detection/ncnn_conversion.py - Camera Test · Standalone camera validation path for vision input debugging.
src/tests/camera_test.py - Motor Test · Standalone motor validation path for motion debugging.
src/tests/motor_test.py - Sensor Test · Standalone sensor validation path for hardware feedback debugging.
src/tests/sensor_test.py
Firmware
Embedded controller logic for motors and sensors.
- Arduino · Embedded controller firmware for actuators and sensors.
src/pickup-arduino/autonomous.ino
Data & models
Training datasets, exported weights, and generated inference artifacts.
- TorchScript · Serialized model artifact for Python-side optimized inference.
src/object-detection/yolo11n.torchscript - NCNN Model · Edge-friendly NCNN export package used for deployment or validation.
src/yolo/yolo11n_ncnn_model - Best Weights · Primary trained YOLO checkpoint used for detection performance.
src/object-detection/best.pt - Base Weights · Ultralytics YOLO base checkpoint used for inference or fine-tuning.
src/object-detection/yolo11n.pt - YOLO Weights · Repository-level YOLO checkpoint used by the alternate YOLO workspace.
src/yolo/yolo11n.pt - World Weights · Additional YOLO world-model checkpoints used for experimentation.
src/object-detection/yolov8l-world.pt - More Weights · Additional YOLO world-model checkpoints used for experimentation.
src/object-detection/yolov8m-world.pt - YOLOv8n · Lightweight YOLO checkpoint available for testing and comparison.
src/object-detection/yolov8n.pt - YOLOv8s World · World-model checkpoint variant for experimentation.
src/object-detection/yolov8s-world.pt - World v2 · Second world-model variant for experimentation.
src/object-detection/yolov8s-worldv2.pt - Obj Dataset · Primary object-detection dataset and label configuration.
src/object-detection/data.yaml - Lego Dataset · Prepared YOLO dataset split with images and labels for training and validation.
src/yolo/Lego/data.yaml - Predict Runs · Saved local prediction outputs from inference experiments.
src/yolo/runs/detect/predict
Connections 20
- Camera Test → Detect: feeds · Camera validation supports the perception pipeline.
- Detect → Autonomous: detections · Inference outputs drive higher-level behavior.
- YOLO Script → Detect: supports · Auxiliary YOLO logic informs the main inference path.
- Learn → Autonomous: experiments · Learning/testing work informs autonomy behavior.
- Autonomous → Arduino: commands · Autonomy sends actuation commands to embedded control.
- Arduino → Sensor Test: feedback · Embedded feedback path is validated through sensor testing.
- Arduino → Motor Test: actuation · Motor control is validated against the firmware boundary.
- Arduino → Sensor Test: sensing · Sensor input is owned by the firmware boundary.
- Obj Dataset → Best Weights: trains · Primary dataset produces the main trained checkpoint.
- Lego Dataset → YOLO Weights: trains · Prepared dataset supports alternate YOLO training artifacts.
- Best Weights → TorchScript: exports · Trained weights are converted into TorchScript.
- Best Weights → NCNN Model: exports · Trained weights are converted into NCNN for deployment.
- Convert → NCNN Model: produces · Conversion script generates the NCNN artifact bundle.
- Base Weights → TorchScript: basis · Base checkpoint can be used to build exported artifacts.
- YOLOv8n → Detect: inference · Lightweight model option for detection experiments.
- World Weights → Detect: inference · Alternative model checkpoint for detection experiments.
- More Weights → Detect: inference · Alternative model checkpoint for detection experiments.
- YOLOv8s World → Detect: inference · Alternative model checkpoint for detection experiments.
- World v2 → Detect: inference · Alternative model checkpoint for detection experiments.
- TorchScript → Predict Runs: evaluated as · Exported model is exercised in local prediction runs.