# harbor302/abu-robocon-2024 architecture

> Architecture diagram of the GitHub repository harbor302/abu-robocon-2024, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/harbor302/abu-robocon-2024
- Repository: https://github.com/harbor302/abu-robocon-2024

## Overview

1. **Overall shape**
   - This is a robotics perception-and-control repository for ROBOCON 2024, not a general app.
   - The runtime is primarily **Python 3.9 on Ubuntu 22.04**, with several **vendored native SDKs** and **pretrained vision models** checked in.

2. **Top-level execution path**
   - The main integration point is **`newmain.py`**.
   - Supporting entrypoints such as **`test.py`** and **`testmain.py`** appear to be validation or bring-up scripts rather than the production control loop.

3. **Primary system boundary: robot application logic**
   - The core robot behavior is split across **`Utils/`** and the root-level main scripts.
   - This boundary likely contains:
     - sensor acquisition,
     - perception,
     - state/decision logic,
     - actuation/serial communication,
     - calibration and coordinate transforms.

4. **Camera and sensor acquisition layer**
   - The repo supports two camera families:
     - **Orbbec Astra depth cameras** under **`Camera/`** and **`OpenNI2/`**
     - **MindVision industrial cameras** under **`Mindvision/`**
   - These directories include:
     - vendor SDK headers and libraries,
     - udev rules,
     - driver configs,
     - sample/demo code.
   - This is a major architectural boundary: the project depends on external camera SDKs rather than pure Python camera libraries.

5. **Perception pipeline**
   - Vision inference is implemented through multiple YOLO/OpenVINO paths:
     - **`Utils/new_vino/new_vinodetect.py`** for OpenVINO-accelerated inference,
     - **`Utils/yolo_openvino/`** for OpenVINO model assets and helper code,
     - **`Utils/yolov5_master/`** as a vendored YOLOv5 codebase with OpenVINO/export support.
   - Model artifacts are stored in **`Models/`** and also in utility-local model folders.
   - This suggests the architecture can switch between native PyTorch/ONNX and OpenVINO deployment paths, with OpenVINO used for production acceleration.

6. **Model assets and deployment artifacts**
   - The repository contains multiple trained model versions:
     - `.pt` and `.onnx` files,
     - OpenVINO exports (`.xml`, `.bin`, `.yaml`),
     - additional mapping files.
   - These are not source code; they are deployable assets consumed by the inference modules.
   - The presence of several dated model folders indicates iterative model training and A/B testing across seasons or runs.

7. **Decision-making / policy subsystem**
   - **`Utils/policy/`** is a separate decision layer based on **TD learning / reinforcement learning**, per the README.
   - It contains:
     - state generation,
     - training,
     - simulation tests,
     - a runtime entry module.
   - Its persisted artifacts include **`all_states.pickle`** and **`policy.bin`**, so the robot’s high-level action selection is data-driven and separated from low-level perception.

8. **Robot control and hardware I/O**
   - **`Utils/serialTools/`** contains RS485 and USB-to-TTL helpers, indicating direct serial control of actuators or embedded controllers.
   - This is the actuation boundary: perception and policy output are likely converted into serial commands here.
   - The hardware abstraction is tied to physical robot communication rather than network services.

9. **Calibration and coordinate conversion**
   - Several files indicate camera-to-robot spatial alignment:
     - **`Camera/calibrate.py`**
     - **`Utils/cam2robot.py`**
     - **`Utils/axis_dot.py`**
     - **`R1/Alignment assistance.py`**
     - depth camera helpers under **`Utils/cam_for_depth/`**
   - These modules likely bridge pixel/depth observations to robot/world coordinates, which is essential for pick-and-place tasks.

10. **Utility and data-handling layer**
   - **`Utils/`** also contains support modules for:
     - frame extraction,
     - video capture/recording,
     - filename management,
     - JSON/LabelMe conversion,
     - navigation and target/state helpers.
   - The checked-in **`Camera/Data/`** and **`Mindvision/demo/python_demo/Camera/Data/`** folders suggest recorded calibration or sample data used offline for testing and tuning.

11. **Development and experiment branches**
   - The **`R1/`** directory contains older or experiment-specific scripts.
   - These likely represent an earlier alignment or control branch, useful for regression testing and prototyping but not the main execution path.

12. **External dependencies and infrastructure**
   - The architecture depends on installed vendor components rather than containerized services:
     - **OpenNI2 / Orbbec drivers**
     - **MindVision SDK**
     - **OpenVINO Toolkit**
     - **Git LFS** for large policy artifacts
   - The included udev rules and install scripts show this repo expects direct access to USB devices on the robot computer.

13. **How data flows through the system**
   - A typical flow is:
     - camera frames / depth data enter through vendor SDKs,
     - vision modules detect balls/frames,
     - calibration utilities convert detections into robot coordinates,
     - policy logic selects actions,
     - serial tools send commands to the robot controller.
   - Offline model files and calibration logs support this loop but are not part of the live control path.

14. **What to draw as the main architecture boundary**
   - Treat this repo as three main blocks:
     - **Sensor acquisition**: Orbbec + MindVision SDK layers
     - **Perception and decision**: YOLO/OpenVINO + RL policy + calibration utilities
     - **Robot actuation**: RS485/USB serial control
   - Around them sits a large support layer of model files, vendor SDKs, and experimental/test scripts.

Main entry points: [`newmain.py`](https://github.com/harbor302/abu-robocon-2024/blob/main/newmain.py) (newmain.py), [`Utils/new_vino/new_vinodetect.py`](https://github.com/harbor302/abu-robocon-2024/blob/main/Utils/new_vino/new_vinodetect.py) (OpenVINO detect), [`Utils/policy/main.py`](https://github.com/harbor302/abu-robocon-2024/blob/main/Utils/policy/main.py) (Policy runtime), [`Utils/cam2robot.py`](https://github.com/harbor302/abu-robocon-2024/blob/main/Utils/cam2robot.py) (cam2robot)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entrypoints"]
  node_newmain["newmain.py<br/>main runtime<br/>[newmain.py]"]
  node_testpy["test.py<br/>bring-up script<br/>[test.py]"]
  node_testmain["testmain.py<br/>bring-up script<br/>[testmain.py]"]
end

subgraph group_sensors["Sensors"]
  node_orbbec_sdk[("Orbbec SDK<br/>depth camera SDK")]
  node_mindvision_sdk[("MindVision SDK<br/>industrial camera SDK")]
  node_astra_depth["Astra depth<br/>depth capture<br/>[Astra.py]"]
  node_camera_util["Camera util<br/>capture helpers<br/>[cap_camera.py]"]
end

subgraph group_perception["Perception"]
  node_vino_detect["OpenVINO detect<br/>inference path<br/>[new_vinodetect.py]"]
  node_yolo_vino["YOLO OpenVINO<br/>model runtime<br/>[vinodetect.py]"]
  node_yolov5_core[("YOLOv5 core<br/>detector stack")]
end

subgraph group_policy["Policy"]
  node_policy_main["Policy runtime<br/>decision layer<br/>[main.py]"]
  node_policy_train["Policy train<br/>training pipeline<br/>[train.py]"]
end

subgraph group_control["Control"]
  node_serial_io["Serial I/O<br/>actuation link<br/>[rs485.py]"]
end

subgraph group_calib["Calibration"]
  node_cam2robot["cam2robot<br/>geometry transform<br/>[cam2robot.py]"]
  node_axisdot["axis_dot<br/>geometry helper<br/>[axis_dot.py]"]
  node_cam_calib["calibrate.py<br/>calibration script<br/>[calibrate.py]"]
end

subgraph group_assets["Assets"]
  node_model_assets[("Models")]
  node_policy_store[("Policy artifacts<br/>persisted policy")]
end

node_newmain -->|"reads frames"| node_astra_depth
node_newmain -->|"captures video"| node_camera_util
node_newmain -->|"runs detection"| node_vino_detect
node_newmain -->|"selects action"| node_policy_main
node_newmain -->|"uses pose"| node_cam2robot
node_newmain -->|"sends commands"| node_serial_io
node_astra_depth -->|"depends on"| node_orbbec_sdk
node_camera_util -->|"depends on"| node_mindvision_sdk
node_vino_detect -->|"loads weights"| node_model_assets
node_vino_detect -->|"shares runtime"| node_yolo_vino
node_yolo_vino -->|"built from"| node_yolov5_core
node_yolov5_core -->|"exports/loads"| node_model_assets
node_policy_main -->|"loads policy"| node_policy_store
node_policy_train -->|"produces"| node_policy_store
node_policy_main -->|"issues actuation"| node_serial_io
node_cam_calib -->|"feeds calibration"| node_cam2robot
node_cam2robot -->|"uses math"| node_axisdot
node_testpy -.->|"brings up"| node_newmain
node_testmain -.->|"mirrors"| node_newmain

click node_newmain "https://github.com/harbor302/abu-robocon-2024/blob/main/newmain.py"
click node_testpy "https://github.com/harbor302/abu-robocon-2024/blob/main/test.py"
click node_testmain "https://github.com/harbor302/abu-robocon-2024/blob/main/testmain.py"
click node_orbbec_sdk "https://github.com/harbor302/abu-robocon-2024/tree/main/Camera/sdk"
click node_mindvision_sdk "https://github.com/harbor302/abu-robocon-2024/tree/main/Mindvision"
click node_astra_depth "https://github.com/harbor302/abu-robocon-2024/blob/main/Utils/cam_for_depth/Astra.py"
click node_camera_util "https://github.com/harbor302/abu-robocon-2024/blob/main/Utils/cap_camera.py"
click node_vino_detect "https://github.com/harbor302/abu-robocon-2024/blob/main/Utils/new_vino/new_vinodetect.py"
click node_yolo_vino "https://github.com/harbor302/abu-robocon-2024/blob/main/Utils/yolo_openvino/vinodetect.py"
click node_yolov5_core "https://github.com/harbor302/abu-robocon-2024/tree/main/Utils/yolov5_master"
click node_model_assets "https://github.com/harbor302/abu-robocon-2024/tree/main/Models"
click node_policy_main "https://github.com/harbor302/abu-robocon-2024/blob/main/Utils/policy/main.py"
click node_policy_train "https://github.com/harbor302/abu-robocon-2024/blob/main/Utils/policy/train.py"
click node_policy_store "https://github.com/harbor302/abu-robocon-2024/tree/main/Utils/policy/models"
click node_cam2robot "https://github.com/harbor302/abu-robocon-2024/blob/main/Utils/cam2robot.py"
click node_axisdot "https://github.com/harbor302/abu-robocon-2024/blob/main/Utils/axis_dot.py"
click node_cam_calib "https://github.com/harbor302/abu-robocon-2024/blob/main/Camera/calibrate.py"
click node_serial_io "https://github.com/harbor302/abu-robocon-2024/blob/main/Utils/serialTools/rs485.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_newmain,node_testpy,node_testmain,node_model_assets,node_policy_store toneBlue
class node_orbbec_sdk,node_mindvision_sdk,node_astra_depth,node_camera_util toneAmber
class node_vino_detect,node_yolo_vino,node_yolov5_core toneMint
class node_policy_main,node_policy_train toneRose
class node_serial_io toneIndigo
class node_cam2robot,node_axisdot,node_cam_calib toneTeal
```

## Components

### Entrypoints

Main runtime and bring-up scripts

- [newmain.py](https://github.com/harbor302/abu-robocon-2024/blob/main/newmain.py): main runtime · Primary integration point for the robot loop · `newmain.py`
- [test.py](https://github.com/harbor302/abu-robocon-2024/blob/main/test.py): bring-up script · Top-level validation or experiment entrypoint · `test.py`
- [testmain.py](https://github.com/harbor302/abu-robocon-2024/blob/main/testmain.py): bring-up script · Alternate main for testing and diagnostics · `testmain.py`

### Sensors

Camera acquisition and vendor SDK layers

- [Orbbec SDK](https://github.com/harbor302/abu-robocon-2024/tree/main/Camera/sdk): depth camera SDK · Vendored OpenNI2/Orbbec headers, drivers, and rules · `Camera/sdk`
- [MindVision SDK](https://github.com/harbor302/abu-robocon-2024/tree/main/Mindvision): industrial camera SDK · Vendored MindVision drivers, headers, and demos · `Mindvision`
- [Astra depth](https://github.com/harbor302/abu-robocon-2024/blob/main/Utils/cam_for_depth/Astra.py): depth capture · Depth-camera helper layer built around OpenNI2 · `Utils/cam_for_depth/Astra.py`
- [Camera util](https://github.com/harbor302/abu-robocon-2024/blob/main/Utils/cap_camera.py): capture helpers · Shared camera acquisition and recording helpers · `Utils/cap_camera.py`

### Perception

Detection and inference paths

- [OpenVINO detect](https://github.com/harbor302/abu-robocon-2024/blob/main/Utils/new_vino/new_vinodetect.py): inference path · OpenVINO-accelerated detection entry module · `Utils/new_vino/new_vinodetect.py`
- [YOLO OpenVINO](https://github.com/harbor302/abu-robocon-2024/blob/main/Utils/yolo_openvino/vinodetect.py): model runtime · OpenVINO model assets and helper code · `Utils/yolo_openvino/vinodetect.py`
- [YOLOv5 core](https://github.com/harbor302/abu-robocon-2024/tree/main/Utils/yolov5_master): detector stack · Vendored YOLOv5 codebase with export and OpenVINO support · `Utils/yolov5_master`

### Policy

State-based decision layer

- [Policy runtime](https://github.com/harbor302/abu-robocon-2024/blob/main/Utils/policy/main.py): decision layer · TD-learning / reinforcement-learning action selection · `Utils/policy/main.py`
- [Policy train](https://github.com/harbor302/abu-robocon-2024/blob/main/Utils/policy/train.py): training pipeline · State generation and policy training scripts · `Utils/policy/train.py`

### Control

Robot command output

- [Serial I/O](https://github.com/harbor302/abu-robocon-2024/blob/main/Utils/serialTools/rs485.py): actuation link · RS485 and USB-to-TTL command helpers · `Utils/serialTools/rs485.py`

### Calibration

Coordinate transforms and alignment

- [cam2robot](https://github.com/harbor302/abu-robocon-2024/blob/main/Utils/cam2robot.py): geometry transform · Maps camera coordinates into robot space · `Utils/cam2robot.py`
- [axis_dot](https://github.com/harbor302/abu-robocon-2024/blob/main/Utils/axis_dot.py): geometry helper · Coordinate and axis utility for alignment math · `Utils/axis_dot.py`
- [calibrate.py](https://github.com/harbor302/abu-robocon-2024/blob/main/Camera/calibrate.py): calibration script · Camera calibration and alignment workflow · `Camera/calibrate.py`

### Assets

Model and persistence artifacts

- [Models](https://github.com/harbor302/abu-robocon-2024/tree/main/Models): trained models · Checked-in .pt/.onnx/OpenVINO model variants · `Models`
- [Policy artifacts](https://github.com/harbor302/abu-robocon-2024/tree/main/Utils/policy/models): persisted policy · Saved states and policy binary used at runtime · `Utils/policy/models`

## Connections

- newmain.py → Astra depth: reads frames · Main loop consumes depth-camera input
- newmain.py → Camera util: captures video · Shared capture and recording utilities support runtime
- newmain.py → OpenVINO detect: runs detection · Primary perception inference path
- newmain.py → Policy runtime: selects action · High-level decision layer chooses robot behavior
- newmain.py → cam2robot: uses pose · Converts detections to robot coordinates
- newmain.py → Serial I/O: sends commands · Outputs motion or actuator commands to hardware
- Astra depth → Orbbec SDK: depends on · Depth acquisition is backed by Orbbec/OpenNI2
- Camera util → MindVision SDK: depends on · Industrial camera capture uses MindVision vendor SDK
- OpenVINO detect → Models: loads weights · Inference consumes deployed model files
- OpenVINO detect → YOLO OpenVINO: shares runtime · OpenVINO detection path reuses model helpers
- YOLO OpenVINO → YOLOv5 core: built from · OpenVINO helpers sit on top of vendored YOLOv5 code
- YOLOv5 core → Models: exports/loads · YOLOv5 code trains, exports, and loads model variants
- Policy runtime → Policy artifacts: loads policy · Runtime policy consumes saved state tables and binary policy
- Policy train → Policy artifacts: produces · Training writes the persisted policy artifacts
- Policy runtime → Serial I/O: issues actuation · Policy decisions become serial commands
- calibrate.py → cam2robot: feeds calibration · Calibration data supports camera-to-robot transforms
- cam2robot → axis_dot: uses math · Coordinate conversion relies on geometry helpers
- test.py → newmain.py: brings up · Validation script exercises the main runtime
- testmain.py → newmain.py: mirrors · Alternative entrypoint for testing the same control flow