# nsu-capstone-2026/nsu_2026capstone architecture

> Architecture diagram of the GitHub repository nsu-capstone-2026/nsu_2026capstone, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nsu-capstone-2026/nsu_2026capstone
- Repository: https://github.com/nsu-capstone-2026/nsu_2026capstone

## Overview

1. **What this repository is**
   - This is a multi-device robotics/vision system that combines:
     - a **Python pose-estimation backend**,
     - a **browser-based control UI**,
     - **Arduino servo control**,
     - **ESP32-CAM / thermal camera streams**,
     - and **ArUco-based calibration/mapping** for 3D tracking.

2. **Top-level execution model**
   - `run.py` is the unified entry point.
   - It can launch:
     - **web only**,
     - **pose only**,
     - or **both together**.
   - The default operational flow is: user opens the setup UI, selects devices, applies settings, then the pose runtime starts and the main dashboard loads.

3. **Frontend boundary**
   - The `web/` folder is a static browser UI.
   - It has two main screens:
     - `setup.html`: device selection and configuration.
     - `index.html`: the main `/app` dashboard.
   - JavaScript modules (`setup.js`, `robot_cam.js`, `theme_handler.js`) handle:
     - configuration submission,
     - live preview display,
     - theme switching,
     - and status polling from the backend.

4. **Backend/API boundary**
   - `pose/web_server.py` is the control plane.
   - It exposes HTTP endpoints such as:
     - `/health`
     - `/api/config`
     - `/api/pose-status`
     - `/api/mapping-status`
   - This server bridges the browser UI to the pose runtime and hardware control, including serial communication to the Arduino.

5. **Pose-estimation runtime**
   - `pose/main.py` is the runtime entry for the 3-camera pose pipeline.
   - The core pipeline lives in `pose/core/`:
     - `capture_3cam.py`: camera acquisition and stream handling,
     - `pose_yolo.py`: 2D keypoint detection using the bundled YOLOv8 pose model,
     - `utils_3d.py`: triangulation and 3D angle computation,
     - `config.py`: runtime/device configuration.
   - The output is not just raw detections; it produces a stabilized arm-angle signal used by the UI and Arduino sync.

6. **3D data flow**
   - Each camera produces 2D pose keypoints.
   - The system combines valid camera pairs to triangulate 3D points.
   - It computes the elbow angle from shoulder–elbow–wrist geometry.
   - Instead of trusting a single estimate, it aggregates pairwise angles and applies filtering:
     - pairwise angle results,
     - median-based short-term filtering,
     - exponential smoothing.
   - The final stabilized angle is the one used for display and serial output.

7. **Calibration and mapping subsystem**
   - `pose/tools/` contains the offline/interactive calibration utilities:
     - `calibration_tool.py`
     - `mapper.py`
   - The workflow is triggered from the web UI via an **ArUco mapping** action.
   - Mapping stores calibration artifacts in `pose/data/`, especially:
     - `marker_map.yaml`
     - `camera_pose_map.yaml`
   - These files define the world reference and fixed camera poses, allowing tracking to continue even when markers are removed in a static camera setup.

8. **Persistent runtime state**
   - The `pose/data/` directory is the persistence boundary for calibration and runtime state.
   - It contains:
     - calibration results,
     - marker/camera pose maps,
     - and runtime status artifacts used by the UI.
   - The README also indicates `runtime_status.json` is used for diagnostics such as active views, pair angles, and pose source.

9. **Hardware/device integrations**
   - The project assumes multiple physical devices:
     - **3 cameras** for pose estimation,
     - an **ESP32-CAM** stream at `http://{ip}:81/stream`,
     - an **MLX90640 thermal camera** stream at `http://{ip}:5000/stream`,
     - and an **Arduino** over serial for PTZ/servo control.
   - The UI setup screen is responsible for selecting these device endpoints, and the backend uses those selections to wire the runtime together.

10. **Arduino control path**
   - `Arduino/Arduino_PTZ/` contains firmware for servo motion control.
   - The Python backend sends serial commands of the form `ELBOW n`.
   - The elbow target is derived from the filtered pose angle and mapped into the servo’s safe range.
   - This is a one-way control loop from pose estimation to actuator command.

11. **ESP32 and edge camera firmware**
   - `Arduino/ESP32CAM/` is a separate ESP32-CAM firmware project.
   - It provides the camera streaming endpoint consumed by the web UI and pose system.
   - The presence of `app_httpd.cpp`, camera pin/config headers, and partition config shows this is an embedded streaming service, not just a client.

12. **Thermal camera service**
   - `RASPI/mxl_web.py` appears to be a Raspberry Pi-side web streamer for the MLX90640 thermal sensor.
   - That makes the thermal camera a distinct external stream source rather than part of the main Python backend.

13. **Operational modes and demos**
   - The system supports mixed real-device and demo configurations.
   - DEMO modes let the UI and control flow run without physical hardware.
   - This means the architecture is designed to degrade gracefully when cameras or Arduino are unavailable.

14. **Architectural summary**
   - The repo is organized around a clear split:
     - **browser UI** for user interaction,
     - **Flask/Python backend** for orchestration and API/state,
     - **vision runtime** for pose inference and 3D reconstruction,
     - **embedded firmware** for camera streaming and servo actuation,
     - **calibration tools** for ArUco-based spatial alignment.
   - The main data path is: camera streams → 2D pose detection → 3D reconstruction/filtering → runtime status/UI → Arduino servo command.

Main entry points: [`pose/web_server.py`](https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/pose/web_server.py) (API server), [`pose/main.py`](https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/pose/main.py) (Pose runtime), [`pose/data`](https://github.com/nsu-capstone-2026/nsu_2026capstone/tree/main/pose/data) (Pose data), [`web/src/theme_handler.js`](https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/web/src/theme_handler.js) (Theme handler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_frontend["Browser UI"]
  node_node_setup_html["Setup screen<br/>browser page<br/>[setup.html]"]
  node_node_index_html["Dashboard<br/>browser page<br/>[index.html]"]
  node_node_setup_js["Setup JS<br/>ui logic<br/>[setup.js]"]
  node_node_robot_cam_js["Camera UI<br/>preview/status<br/>[robot_cam.js]"]
  node_node_theme_js["Theme handler<br/>ui chrome<br/>[theme_handler.js]"]
end

subgraph group_grp_backend["Python Orchestration"]
  node_node_run["run.py<br/>launcher<br/>[run.py]"]
  node_node_web_server["API server<br/>flask control plane<br/>[web_server.py]"]
end

subgraph group_grp_pose["Pose Runtime"]
  node_node_main["Pose runtime<br/>pipeline entry<br/>[main.py]"]
  node_node_capture["3-cam capture<br/>camera I/O<br/>[capture_3cam.py]"]
  node_node_pose_yolo["Pose detector<br/>2D keypoints<br/>[pose_yolo.py]"]
  node_node_utils3d["3D solver<br/>triangulation<br/>[utils_3d.py]"]
  node_node_config["Runtime config<br/>device config<br/>[config.py]"]
  node_node_model["YOLO pose model<br/>weights<br/>[yolov8n-pose.pt]"]
end

subgraph group_grp_calibration["Calibration & State"]
  node_node_mapper["Mapper<br/>aruco mapping<br/>[mapper.py]"]
  node_node_calib_tool["Calibrator<br/>offline tool"]
  node_node_data[("Pose data<br/>persistent state")]
end

subgraph group_grp_hardware["Edge Hardware"]
  node_node_arduino_ptz["PTZ firmware<br/>servo control<br/>[Arduino_PTZ.ino]"]
  node_node_esp32cam["ESP32-CAM<br/>streaming firmware<br/>[ESP32CAM.ino]"]
  node_node_mxl_web["Thermal stream<br/>raspi streamer<br/>[mxl_web.py]"]
end

node_node_run -->|"launches"| node_node_web_server
node_node_run -->|"launches"| node_node_main
node_node_web_server -->|"serves"| node_node_setup_html
node_node_web_server -->|"serves"| node_node_index_html
node_node_setup_js -->|"posts config"| node_node_web_server
node_node_robot_cam_js -->|"polls status"| node_node_web_server
node_node_theme_js -->|"updates UI"| node_node_setup_html
node_node_theme_js -->|"updates UI"| node_node_index_html
node_node_web_server -->|"loads config"| node_node_config
node_node_web_server -->|"reads/writes"| node_node_data
node_node_web_server -->|"serial commands"| node_node_arduino_ptz
node_node_main -->|"uses"| node_node_capture
node_node_main -->|"uses"| node_node_pose_yolo
node_node_main -->|"uses"| node_node_utils3d
node_node_pose_yolo -->|"loads"| node_node_model
node_node_utils3d -->|"stores status"| node_node_data
node_node_mapper -->|"writes maps"| node_node_data
node_node_calib_tool -->|"reads/writes"| node_node_data
node_node_web_server -->|"triggers"| node_node_mapper
node_node_web_server -->|"exposes"| node_node_calib_tool
node_node_esp32cam -->|"video source"| node_node_web_server
node_node_mxl_web -->|"thermal source"| node_node_web_server

click node_node_run "https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/run.py"
click node_node_web_server "https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/pose/web_server.py"
click node_node_main "https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/pose/main.py"
click node_node_capture "https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/pose/core/capture_3cam.py"
click node_node_pose_yolo "https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/pose/core/pose_yolo.py"
click node_node_utils3d "https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/pose/core/utils_3d.py"
click node_node_config "https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/pose/core/config.py"
click node_node_setup_html "https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/web/setup.html"
click node_node_index_html "https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/web/index.html"
click node_node_setup_js "https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/web/src/setup.js"
click node_node_robot_cam_js "https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/web/src/robot_cam.js"
click node_node_theme_js "https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/web/src/theme_handler.js"
click node_node_mapper "https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/pose/tools/mapper.py"
click node_node_calib_tool "https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/pose/tools/calibration_tool.py"
click node_node_data "https://github.com/nsu-capstone-2026/nsu_2026capstone/tree/main/pose/data"
click node_node_arduino_ptz "https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/Arduino/Arduino_PTZ/Arduino_PTZ.ino"
click node_node_esp32cam "https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/Arduino/ESP32CAM/ESP32CAM.ino"
click node_node_mxl_web "https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/RASPI/mxl_web.py"
click node_node_model "https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/pose/models/yolov8n-pose.pt"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_setup_html,node_node_index_html,node_node_setup_js,node_node_robot_cam_js,node_node_theme_js toneBlue
class node_node_run,node_node_web_server toneAmber
class node_node_main,node_node_capture,node_node_pose_yolo,node_node_utils3d,node_node_config,node_node_model toneMint
class node_node_mapper,node_node_calib_tool,node_node_data toneRose
class node_node_arduino_ptz,node_node_esp32cam,node_node_mxl_web toneIndigo
```

## Components

### Browser UI

Static setup and dashboard screens served to the browser.

- [Setup screen](https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/web/setup.html): browser page · Device selection and configuration UI. · `web/setup.html`
- [Dashboard](https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/web/index.html): browser page · Main application dashboard loaded at /app. · `web/index.html`
- [Setup JS](https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/web/src/setup.js): ui logic · Submits configuration and coordinates setup interactions. · `web/src/setup.js`
- [Camera UI](https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/web/src/robot_cam.js): preview/status · Handles live preview display and backend polling. · `web/src/robot_cam.js`
- [Theme handler](https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/web/src/theme_handler.js): ui chrome · Manages browser theme switching. · `web/src/theme_handler.js`

### Python Orchestration

HTTP control plane and unified runtime entrypoints.

- [run.py](https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/run.py): launcher · Unified entrypoint that starts web only, pose only, or both together. · `run.py`
- [API server](https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/pose/web_server.py): flask control plane · Backend HTTP boundary exposing config, status, and health endpoints. · `pose/web_server.py`

### Pose Runtime

3-camera capture, pose inference, reconstruction, and filtering.

- [Pose runtime](https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/pose/main.py): pipeline entry · Entry for the live pose-estimation and tracking runtime. · `pose/main.py`
- [3-cam capture](https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/pose/core/capture_3cam.py): camera I/O · Acquires and manages the multi-camera video streams. · `pose/core/capture_3cam.py`
- [Pose detector](https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/pose/core/pose_yolo.py): 2D keypoints · Runs YOLOv8 pose inference to extract 2D human keypoints. · `pose/core/pose_yolo.py`
- [3D solver](https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/pose/core/utils_3d.py): triangulation · Combines camera pairs into 3D points and computes arm angles. · `pose/core/utils_3d.py`
- [Runtime config](https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/pose/core/config.py): device config · Shared runtime and device configuration for the pose pipeline. · `pose/core/config.py`
- [YOLO pose model](https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/pose/models/yolov8n-pose.pt): weights · Bundled YOLOv8n pose weights used by the detector. · `pose/models/yolov8n-pose.pt`

### Calibration & State

ArUco mapping tools and persisted runtime/calibration artifacts.

- [Mapper](https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/pose/tools/mapper.py): aruco mapping · Interactive mapping utility that builds camera and marker pose references. · `pose/tools/mapper.py`
- [Calibrator](https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/pose/tools/calibration_tool.py): offline tool · Calibration utility for alignment and verification workflows. · `pose/tools/calibration_tool.py`
- [Pose data](https://github.com/nsu-capstone-2026/nsu_2026capstone/tree/main/pose/data): persistent state · Stores calibration results, marker maps, camera pose maps, and runtime status artifacts. · `pose/data`

### Edge Hardware

Embedded camera streaming and servo actuation firmware.

- [PTZ firmware](https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/Arduino/Arduino_PTZ/Arduino_PTZ.ino): servo control · Arduino firmware that receives elbow targets and drives servos. · `Arduino/Arduino_PTZ/Arduino_PTZ.ino`
- [ESP32-CAM](https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/Arduino/ESP32CAM/ESP32CAM.ino): streaming firmware · Embedded camera streamer that exposes a network video endpoint. · `Arduino/ESP32CAM/ESP32CAM.ino`
- [Thermal stream](https://github.com/nsu-capstone-2026/nsu_2026capstone/blob/main/RASPI/mxl_web.py): raspi streamer · Raspberry Pi streamer for the MLX90640 thermal camera. · `RASPI/mxl_web.py`

## Connections

- run.py → API server: launches · Starts the control plane when web or combined mode is selected.
- run.py → Pose runtime: launches · Starts the pose runtime when pose or combined mode is selected.
- API server → Setup screen: serves · Delivers the setup page to the browser.
- API server → Dashboard: serves · Delivers the main dashboard page.
- Setup JS → API server: posts config · Sends device and runtime settings to the backend API.
- Camera UI → API server: polls status · Reads pose and mapping status plus health information.
- Theme handler → Setup screen: updates UI · Controls theme presentation in the browser.
- Theme handler → Dashboard: updates UI · Controls theme presentation in the browser.
- API server → Runtime config: loads config · Uses selected devices and runtime settings to configure the system.
- API server → Pose data: reads/writes · Persists and retrieves runtime status and calibration artifacts.
- API server → PTZ firmware: serial commands · Sends servo targets derived from stabilized arm angle.
- Pose runtime → 3-cam capture: uses · Pulls synchronized frames from the configured cameras.
- Pose runtime → Pose detector: uses · Runs keypoint detection on captured frames.
- Pose runtime → 3D solver: uses · Reconstructs 3D pose and computes stabilized angles.
- Pose detector → YOLO pose model: loads · Uses the bundled pose weights for inference.
- 3D solver → Pose data: stores status · Writes diagnostics and pose artifacts used by the UI.
- Mapper → Pose data: writes maps · Produces marker and camera pose maps during calibration.
- Calibrator → Pose data: reads/writes · Uses persisted calibration artifacts for offline workflows.
- API server → Mapper: triggers · Starts ArUco mapping from the browser workflow.
- API server → Calibrator: exposes · Makes calibration utilities available through the backend control flow.
- ESP32-CAM → API server: video source · Provides a network camera stream consumed by the system.
- Thermal stream → API server: thermal source · Provides the thermal camera stream consumed by the system.
