How harbor302/abu-robocon-2024 works
1. Overall shape
- This is a robotics perception-and-control repository for ROBOCON 2024, not a general app.
- The runtime is primarily Python 3.9 on Ubuntu 22.04, with several vendored native SDKs and pretrained vision models checked in.
2. Top-level execution path
- The main integration point is newmain.py.
- Supporting entrypoints such as test.py and testmain.py appear to be validation or bring-up scripts rather than the production control loop.
3. Primary system boundary: robot application logic
- The core robot behavior is split across Utils/ and the root-level main scripts.
- This boundary likely contains:
- sensor acquisition,
- perception,
- state/decision logic,
- actuation/serial communication,
- calibration and coordinate transforms.
4. Camera and sensor acquisition layer
- The repo supports two camera families:
- Orbbec Astra depth cameras under Camera/ and OpenNI2/
- MindVision industrial cameras under Mindvision/
- These directories include:
- vendor SDK headers and libraries,
- udev rules,
- driver configs,
- sample/demo code.
- This is a major architectural boundary: the project depends on external camera SDKs rather than pure Python camera libraries.
5. Perception pipeline
- Vision inference is implemented through multiple YOLO/OpenVINO paths:
- Utils/new_vino/new_vinodetect.py for OpenVINO-accelerated inference,
- Utils/yolo_openvino/ for OpenVINO model assets and helper code,
- Utils/yolov5_master/ as a vendored YOLOv5 codebase with OpenVINO/export support.
- Model artifacts are stored in Models/ and also in utility-local model folders.
- This suggests the architecture can switch between native PyTorch/ONNX and OpenVINO deployment paths, with OpenVINO used for production acceleration.
6. Model assets and deployment artifacts
- The repository contains multiple trained model versions:
- .pt and .onnx files,
- OpenVINO exports (.xml, .bin, .yaml),
- additional mapping files.
- These are not source code; they are deployable assets consumed by the inference modules.
- The presence of several dated model folders indicates iterative model training and A/B testing across seasons or runs.
7. Decision-making / policy subsystem
- Utils/policy/ is a separate decision layer based on TD learning / reinforcement learning, per the README.
- It contains:
- state generation,
- training,
- simulation tests,
- a runtime entry module.
- Its persisted artifacts include all_states.pickle and policy.bin, so the robot’s high-level action selection is data-driven and separated from low-level perception.
8. Robot control and hardware I/O
- Utils/serialTools/ contains RS485 and USB-to-TTL helpers, indicating direct serial control of actuators or embedded controllers.
- This is the actuation boundary: perception and policy output are likely converted into serial commands here.
- The hardware abstraction is tied to physical robot communication rather than network services.
9. Calibration and coordinate conversion
- Several files indicate camera-to-robot spatial alignment:
- Camera/calibrate.py
- Utils/cam2robot.py
- Utils/axis_dot.py
- R1/Alignment assistance.py
- depth camera helpers under Utils/cam_for_depth/
- These modules likely bridge pixel/depth observations to robot/world coordinates, which is essential for pick-and-place tasks.
10. Utility and data-handling layer
- Utils/ also contains support modules for:
- frame extraction,
- video capture/recording,
- filename management,
- JSON/LabelMe conversion,
- navigation and target/state helpers.
- The checked-in Camera/Data/ and Mindvision/demo/python_demo/Camera/Data/ folders suggest recorded calibration or sample data used offline for testing and tuning.
11. Development and experiment branches
- The R1/ directory contains older or experiment-specific scripts.
- These likely represent an earlier alignment or control branch, useful for regression testing and prototyping but not the main execution path.
12. External dependencies and infrastructure
- The architecture depends on installed vendor components rather than containerized services:
- OpenNI2 / Orbbec drivers
- MindVision SDK
- OpenVINO Toolkit
- Git LFS for large policy artifacts
- The included udev rules and install scripts show this repo expects direct access to USB devices on the robot computer.
13. How data flows through the system
- A typical flow is:
- camera frames / depth data enter through vendor SDKs,
- vision modules detect balls/frames,
- calibration utilities convert detections into robot coordinates,
- policy logic selects actions,
- serial tools send commands to the robot controller.
- Offline model files and calibration logs support this loop but are not part of the live control path.
14. What to draw as the main architecture boundary
- Treat this repo as three main blocks:
- Sensor acquisition: Orbbec + MindVision SDK layers
- Perception and decision: YOLO/OpenVINO + RL policy + calibration utilities
- Robot actuation: RS485/USB serial control
- Around them sits a large support layer of model files, vendor SDKs, and experimental/test scripts.
Main entry points: newmain.py newmain.py · new_vinodetect.py OpenVINO detect · main.py Policy runtime · cam2robot.py cam2robot
Components 18
Entrypoints
Main runtime and bring-up scripts
- newmain.py · Primary integration point for the robot loop
newmain.py - test.py · Top-level validation or experiment entrypoint
test.py - testmain.py · Alternate main for testing and diagnostics
testmain.py
Sensors
Camera acquisition and vendor SDK layers
- Orbbec SDK · Vendored OpenNI2/Orbbec headers, drivers, and rules
Camera/sdk - MindVision SDK · Vendored MindVision drivers, headers, and demos
Mindvision - Astra depth · Depth-camera helper layer built around OpenNI2
Utils/cam_for_depth/Astra.py - Camera util · Shared camera acquisition and recording helpers
Utils/cap_camera.py
Perception
Detection and inference paths
- OpenVINO detect · OpenVINO-accelerated detection entry module
Utils/new_vino/new_vinodetect.py - YOLO OpenVINO · OpenVINO model assets and helper code
Utils/yolo_openvino/vinodetect.py - YOLOv5 core · Vendored YOLOv5 codebase with export and OpenVINO support
Utils/yolov5_master
Policy
State-based decision layer
- Policy runtime · TD-learning / reinforcement-learning action selection
Utils/policy/main.py - Policy train · State generation and policy training scripts
Utils/policy/train.py
Control
Robot command output
- Serial I/O · RS485 and USB-to-TTL command helpers
Utils/serialTools/rs485.py
Calibration
Coordinate transforms and alignment
- cam2robot · Maps camera coordinates into robot space
Utils/cam2robot.py - axis_dot · Coordinate and axis utility for alignment math
Utils/axis_dot.py - calibrate.py · Camera calibration and alignment workflow
Camera/calibrate.py
Assets
Model and persistence artifacts
- Models · Checked-in .pt/.onnx/OpenVINO model variants
Models - Policy artifacts · Saved states and policy binary used at runtime
Utils/policy/models
Connections 19
- newmain.py → Astra depth: reads frames · Main loop consumes depth-camera input
- newmain.py → Camera util: captures video · Shared capture and recording utilities support runtime
- newmain.py → OpenVINO detect: runs detection · Primary perception inference path
- newmain.py → Policy runtime: selects action · High-level decision layer chooses robot behavior
- newmain.py → cam2robot: uses pose · Converts detections to robot coordinates
- newmain.py → Serial I/O: sends commands · Outputs motion or actuator commands to hardware
- Astra depth → Orbbec SDK: depends on · Depth acquisition is backed by Orbbec/OpenNI2
- Camera util → MindVision SDK: depends on · Industrial camera capture uses MindVision vendor SDK
- OpenVINO detect → Models: loads weights · Inference consumes deployed model files
- OpenVINO detect → YOLO OpenVINO: shares runtime · OpenVINO detection path reuses model helpers
- YOLO OpenVINO → YOLOv5 core: built from · OpenVINO helpers sit on top of vendored YOLOv5 code
- YOLOv5 core → Models: exports/loads · YOLOv5 code trains, exports, and loads model variants
- Policy runtime → Policy artifacts: loads policy · Runtime policy consumes saved state tables and binary policy
- Policy train → Policy artifacts: produces · Training writes the persisted policy artifacts
- Policy runtime → Serial I/O: issues actuation · Policy decisions become serial commands
- calibrate.py → cam2robot: feeds calibration · Calibration data supports camera-to-robot transforms
- cam2robot → axis_dot: uses math · Coordinate conversion relies on geometry helpers
- test.py → newmain.py: brings up · Validation script exercises the main runtime
- testmain.py → newmain.py: mirrors · Alternative entrypoint for testing the same control flow