How nsu-capstone-2026/nsu_2026capstone works
1. What this repository is
- This is a multi-device robotics/vision system that combines:
- a Python pose-estimation backend,
- a browser-based control UI,
- Arduino servo control,
- ESP32-CAM / thermal camera streams,
- and ArUco-based calibration/mapping for 3D tracking.
2. Top-level execution model
- run.py is the unified entry point.
- It can launch:
- web only,
- pose only,
- or both together.
- The default operational flow is: user opens the setup UI, selects devices, applies settings, then the pose runtime starts and the main dashboard loads.
3. Frontend boundary
- The web/ folder is a static browser UI.
- It has two main screens:
- setup.html: device selection and configuration.
- index.html: the main /app dashboard.
- JavaScript modules (setup.js, robot_cam.js, theme_handler.js) handle:
- configuration submission,
- live preview display,
- theme switching,
- and status polling from the backend.
4. Backend/API boundary
- pose/web_server.py is the control plane.
- It exposes HTTP endpoints such as:
- /health
- /api/config
- /api/pose-status
- /api/mapping-status
- This server bridges the browser UI to the pose runtime and hardware control, including serial communication to the Arduino.
5. Pose-estimation runtime
- pose/main.py is the runtime entry for the 3-camera pose pipeline.
- The core pipeline lives in pose/core/:
- capture_3cam.py: camera acquisition and stream handling,
- pose_yolo.py: 2D keypoint detection using the bundled YOLOv8 pose model,
- utils_3d.py: triangulation and 3D angle computation,
- config.py: runtime/device configuration.
- The output is not just raw detections; it produces a stabilized arm-angle signal used by the UI and Arduino sync.
6. 3D data flow
- Each camera produces 2D pose keypoints.
- The system combines valid camera pairs to triangulate 3D points.
- It computes the elbow angle from shoulder–elbow–wrist geometry.
- Instead of trusting a single estimate, it aggregates pairwise angles and applies filtering:
- pairwise angle results,
- median-based short-term filtering,
- exponential smoothing.
- The final stabilized angle is the one used for display and serial output.
7. Calibration and mapping subsystem
- pose/tools/ contains the offline/interactive calibration utilities:
- calibration_tool.py
- mapper.py
- The workflow is triggered from the web UI via an ArUco mapping action.
- Mapping stores calibration artifacts in pose/data/, especially:
- marker_map.yaml
- camera_pose_map.yaml
- These files define the world reference and fixed camera poses, allowing tracking to continue even when markers are removed in a static camera setup.
8. Persistent runtime state
- The pose/data/ directory is the persistence boundary for calibration and runtime state.
- It contains:
- calibration results,
- marker/camera pose maps,
- and runtime status artifacts used by the UI.
- The README also indicates runtime_status.json is used for diagnostics such as active views, pair angles, and pose source.
9. Hardware/device integrations
- The project assumes multiple physical devices:
- 3 cameras for pose estimation,
- an ESP32-CAM stream at http://{ip}:81/stream,
- an MLX90640 thermal camera stream at http://{ip}:5000/stream,
- and an Arduino over serial for PTZ/servo control.
- The UI setup screen is responsible for selecting these device endpoints, and the backend uses those selections to wire the runtime together.
10. Arduino control path
- Arduino/Arduino_PTZ/ contains firmware for servo motion control.
- The Python backend sends serial commands of the form ELBOW n.
- The elbow target is derived from the filtered pose angle and mapped into the servo’s safe range.
- This is a one-way control loop from pose estimation to actuator command.
11. ESP32 and edge camera firmware
- Arduino/ESP32CAM/ is a separate ESP32-CAM firmware project.
- It provides the camera streaming endpoint consumed by the web UI and pose system.
- The presence of app_httpd.cpp, camera pin/config headers, and partition config shows this is an embedded streaming service, not just a client.
12. Thermal camera service
- RASPI/mxl_web.py appears to be a Raspberry Pi-side web streamer for the MLX90640 thermal sensor.
- That makes the thermal camera a distinct external stream source rather than part of the main Python backend.
13. Operational modes and demos
- The system supports mixed real-device and demo configurations.
- DEMO modes let the UI and control flow run without physical hardware.
- This means the architecture is designed to degrade gracefully when cameras or Arduino are unavailable.
14. Architectural summary
- The repo is organized around a clear split:
- browser UI for user interaction,
- Flask/Python backend for orchestration and API/state,
- vision runtime for pose inference and 3D reconstruction,
- embedded firmware for camera streaming and servo actuation,
- calibration tools for ArUco-based spatial alignment.
- The main data path is: camera streams → 2D pose detection → 3D reconstruction/filtering → runtime status/UI → Arduino servo command.
Main entry points: web_server.py API server · main.py Pose runtime · data Pose data · theme_handler.js Theme handler
Components 19
Browser UI
Static setup and dashboard screens served to the browser.
- Setup screen · Device selection and configuration UI.
web/setup.html - Dashboard · Main application dashboard loaded at /app.
web/index.html - Setup JS · Submits configuration and coordinates setup interactions.
web/src/setup.js - Camera UI · Handles live preview display and backend polling.
web/src/robot_cam.js - Theme handler · Manages browser theme switching.
web/src/theme_handler.js
Python Orchestration
HTTP control plane and unified runtime entrypoints.
- run.py · Unified entrypoint that starts web only, pose only, or both together.
run.py - API server · Backend HTTP boundary exposing config, status, and health endpoints.
pose/web_server.py
Pose Runtime
3-camera capture, pose inference, reconstruction, and filtering.
- Pose runtime · Entry for the live pose-estimation and tracking runtime.
pose/main.py - 3-cam capture · Acquires and manages the multi-camera video streams.
pose/core/capture_3cam.py - Pose detector · Runs YOLOv8 pose inference to extract 2D human keypoints.
pose/core/pose_yolo.py - 3D solver · Combines camera pairs into 3D points and computes arm angles.
pose/core/utils_3d.py - Runtime config · Shared runtime and device configuration for the pose pipeline.
pose/core/config.py - YOLO pose model · Bundled YOLOv8n pose weights used by the detector.
pose/models/yolov8n-pose.pt
Calibration & State
ArUco mapping tools and persisted runtime/calibration artifacts.
- Mapper · Interactive mapping utility that builds camera and marker pose references.
pose/tools/mapper.py - Calibrator · Calibration utility for alignment and verification workflows.
pose/tools/calibration_tool.py - Pose data · Stores calibration results, marker maps, camera pose maps, and runtime status artifacts.
pose/data
Edge Hardware
Embedded camera streaming and servo actuation firmware.
- PTZ firmware · Arduino firmware that receives elbow targets and drives servos.
Arduino/Arduino_PTZ/Arduino_PTZ.ino - ESP32-CAM · Embedded camera streamer that exposes a network video endpoint.
Arduino/ESP32CAM/ESP32CAM.ino - Thermal stream · Raspberry Pi streamer for the MLX90640 thermal camera.
RASPI/mxl_web.py
Connections 22
- run.py → API server: launches · Starts the control plane when web or combined mode is selected.
- run.py → Pose runtime: launches · Starts the pose runtime when pose or combined mode is selected.
- API server → Setup screen: serves · Delivers the setup page to the browser.
- API server → Dashboard: serves · Delivers the main dashboard page.
- Setup JS → API server: posts config · Sends device and runtime settings to the backend API.
- Camera UI → API server: polls status · Reads pose and mapping status plus health information.
- Theme handler → Setup screen: updates UI · Controls theme presentation in the browser.
- Theme handler → Dashboard: updates UI · Controls theme presentation in the browser.
- API server → Runtime config: loads config · Uses selected devices and runtime settings to configure the system.
- API server → Pose data: reads/writes · Persists and retrieves runtime status and calibration artifacts.
- API server → PTZ firmware: serial commands · Sends servo targets derived from stabilized arm angle.
- Pose runtime → 3-cam capture: uses · Pulls synchronized frames from the configured cameras.
- Pose runtime → Pose detector: uses · Runs keypoint detection on captured frames.
- Pose runtime → 3D solver: uses · Reconstructs 3D pose and computes stabilized angles.
- Pose detector → YOLO pose model: loads · Uses the bundled pose weights for inference.
- 3D solver → Pose data: stores status · Writes diagnostics and pose artifacts used by the UI.
- Mapper → Pose data: writes maps · Produces marker and camera pose maps during calibration.
- Calibrator → Pose data: reads/writes · Uses persisted calibration artifacts for offline workflows.
- API server → Mapper: triggers · Starts ArUco mapping from the browser workflow.
- API server → Calibrator: exposes · Makes calibration utilities available through the backend control flow.
- ESP32-CAM → API server: video source · Provides a network camera stream consumed by the system.
- Thermal stream → API server: thermal source · Provides the thermal camera stream consumed by the system.