# 23120340/region_based_convolutional_neural_network_demo architecture

> Architecture diagram of the GitHub repository 23120340/region_based_convolutional_neural_network_demo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/23120340/region_based_convolutional_neural_network_demo
- Repository: https://github.com/23120340/region_based_convolutional_neural_network_demo

## Overview

Earbud Hybrid Assembly Monitor is a camera-based prototype that checks the sequence open case → insert first earbud → insert second earbud → close case and reports PASS, violation, or the next step. The main runtime combines object geometry from YOLO with temporal action recognition, fuses both signals, then checks workflow order. The README also describes a separate action-model data and training workflow. The graph reflects documented behavior and indexed source references; some internal wiring is not visible in the excerpts, so uncertain connections are omitted.

Main entry points: [`scripts/run_hybrid.py`](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/scripts/run_hybrid.py) (Hybrid runtime), [`src/assembly/models/vit_lstm_recognizer.py`](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/models/vit_lstm_recognizer.py) (Temporal action recognizer), [`src/assembly/fsm.py`](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/fsm.py) (Assembly workflow FSM), [`src/assembly/phone_camera.py`](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/phone_camera.py) (Phone stream server)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_capture["Capture and UI"]
  node_runtime["Hybrid runtime<br/>[run_hybrid.py]"]
  node_camera_input["Frame input<br/>[camera_input.py]"]
  node_phone_server["Phone stream server<br/>[phone_camera.py]"]
  node_phone_ui["Phone connection UI<br/>[phone_camera_ui.py]"]
  node_dashboard["Runtime dashboard"]
end

subgraph group_perception["Perception"]
  node_detector["YOLO detector"]
  node_vision["Scene geometry<br/>[vision.py]"]
  node_recognizer["Temporal action recognizer"]
  node_encoder["DINO spatial encoder<br/>[spatial_encoder.py]"]
  node_action_net["BiLSTM classifier<br/>[action_net.py]"]
end

subgraph group_workflow["Fusion and workflow"]
  node_fusion["Slot and action fusion<br/>[slot_fusion.py]"]
  node_earbud_fusion["Earbud fusion logic<br/>[earbud_fusion.py]"]
  node_smoother["Prediction debouncer<br/>[smoother.py]"]
  node_fsm["Assembly workflow FSM<br/>[fsm.py]"]
  node_fsm_config["Workflow configuration<br/>[config.py]"]
  node_monitor["Event monitor<br/>[monitor.py]"]
end

subgraph group_modeldata["Action model development"]
  node_annotations["Action annotations"]
  node_dataset["Action dataset<br/>[action_dataset.py]"]
  node_feature_extraction["Spatial features"]
  node_training["Action model training"]
  node_evaluation["Model evaluation"]
end

node_operator(("Operator"))
node_camera(("Camera or video"))
node_phone(("Phone camera"))
node_result(("PASS, violation, next step"))

node_operator -->|"starts monitor"| node_runtime
node_camera -->|"supplies frames"| node_camera_input
node_phone -->|"streams video"| node_phone_server
node_runtime -->|"reads frames"| node_camera_input
node_runtime -.->|"starts optional stream"| node_phone_server
node_runtime -->|"opens connection UI"| node_phone_ui
node_phone_ui -->|"connects session"| node_phone_server
node_runtime -->|"runs detection"| node_detector
node_detector -->|"produces detections"| node_vision
node_runtime -->|"predicts actions"| node_recognizer
node_recognizer -->|"encodes frames"| node_encoder
node_recognizer -->|"classifies windows"| node_action_net
node_runtime -->|"fuses evidence"| node_fusion
node_fusion -->|"uses earbud rules"| node_earbud_fusion
node_runtime -->|"stabilizes predictions"| node_smoother
node_runtime -->|"submits action events"| node_fsm
node_fsm -->|"loads workflow rules"| node_fsm_config
node_runtime -->|"updates event monitor"| node_monitor
node_runtime -->|"updates display"| node_dashboard
node_dashboard -->|"shows status"| node_operator
node_fsm -->|"returns outcome"| node_result
node_annotations -.->|"labels action data"| node_dataset
node_dataset -.->|"provides video data"| node_feature_extraction
node_feature_extraction -.->|"supplies features"| node_training
node_training -.->|"evaluates model"| node_evaluation

click node_runtime "https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/scripts/run_hybrid.py"
click node_camera_input "https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/camera_input.py"
click node_phone_server "https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/phone_camera.py"
click node_phone_ui "https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/phone_camera_ui.py"
click node_dashboard "https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/hybrid_dashboard.py"
click node_detector "https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/yolo_world_detector.py"
click node_vision "https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/vision.py"
click node_recognizer "https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/models/vit_lstm_recognizer.py"
click node_encoder "https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/models/spatial_encoder.py"
click node_action_net "https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/models/action_net.py"
click node_fusion "https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/slot_fusion.py"
click node_earbud_fusion "https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/earbud_fusion.py"
click node_smoother "https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/smoother.py"
click node_fsm "https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/fsm.py"
click node_fsm_config "https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/config.py"
click node_monitor "https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/monitor.py"
click node_annotations "https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/scripts/annotate_actions.py"
click node_dataset "https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/action_dataset.py"
click node_feature_extraction "https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/scripts/extract_spatial_features.py"
click node_training "https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/scripts/train_action_model.py"
click node_evaluation "https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/scripts/evaluate_action_model.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runtime,node_camera_input,node_phone_server,node_phone_ui,node_dashboard toneBlue
class node_detector,node_vision,node_recognizer,node_encoder,node_action_net toneAmber
class node_fusion,node_earbud_fusion,node_smoother,node_fsm,node_fsm_config,node_monitor toneMint
class node_annotations,node_dataset,node_feature_extraction,node_training,node_evaluation toneRose
class node_operator,node_camera,node_phone,node_result toneIndigo
```

## Components

### Capture and UI

- [Hybrid runtime](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/scripts/run_hybrid.py): `scripts/run_hybrid.py`
- [Frame input](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/camera_input.py): `src/assembly/camera_input.py`
- [Phone stream server](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/phone_camera.py): `src/assembly/phone_camera.py`
- [Phone connection UI](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/phone_camera_ui.py): `src/assembly/phone_camera_ui.py`
- [Runtime dashboard](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/hybrid_dashboard.py): `src/assembly/hybrid_dashboard.py`

### Perception

- [YOLO detector](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/yolo_world_detector.py): `src/assembly/yolo_world_detector.py`
- [Scene geometry](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/vision.py): `src/assembly/vision.py`
- [Temporal action recognizer](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/models/vit_lstm_recognizer.py): `src/assembly/models/vit_lstm_recognizer.py`
- [DINO spatial encoder](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/models/spatial_encoder.py): `src/assembly/models/spatial_encoder.py`
- [BiLSTM classifier](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/models/action_net.py): `src/assembly/models/action_net.py`

### Fusion and workflow

- [Slot and action fusion](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/slot_fusion.py): `src/assembly/slot_fusion.py`
- [Earbud fusion logic](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/earbud_fusion.py): `src/assembly/earbud_fusion.py`
- [Prediction debouncer](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/smoother.py): `src/assembly/smoother.py`
- [Assembly workflow FSM](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/fsm.py): `src/assembly/fsm.py`
- [Workflow configuration](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/config.py): `src/assembly/config.py`
- [Event monitor](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/monitor.py): `src/assembly/monitor.py`

### Action model development

- [Action annotations](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/scripts/annotate_actions.py): `scripts/annotate_actions.py`
- [Action dataset](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/src/assembly/action_dataset.py): `src/assembly/action_dataset.py`
- [Spatial features](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/scripts/extract_spatial_features.py): `scripts/extract_spatial_features.py`
- [Action model training](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/scripts/train_action_model.py): `scripts/train_action_model.py`
- [Model evaluation](https://github.com/23120340/region_based_convolutional_neural_network_demo/blob/main/scripts/evaluate_action_model.py): `scripts/evaluate_action_model.py`

### Other components

- **Operator**
- **Camera or video**
- **Phone camera**
- **PASS, violation, next step**

## Connections

- Operator → Hybrid runtime: starts monitor (evidence: `README.md`)
- Camera or video → Frame input: supplies frames (evidence: `README.md`)
- Phone camera → Phone stream server: streams video (evidence: `README.md`)
- Hybrid runtime → Frame input: reads frames (evidence: `scripts/run_hybrid.py`)
- Hybrid runtime → Phone stream server: starts optional stream (evidence: `scripts/run_hybrid.py`)
- Hybrid runtime → Phone connection UI: opens connection UI (evidence: `scripts/run_hybrid.py`)
- Phone connection UI → Phone stream server: connects session (evidence: `src/assembly/phone_camera_ui.py`)
- Hybrid runtime → YOLO detector: runs detection (evidence: `scripts/run_hybrid.py`)
- YOLO detector → Scene geometry: produces detections (evidence: `src/assembly/yolo_world_detector.py`)
- Hybrid runtime → Temporal action recognizer: predicts actions (evidence: `scripts/run_hybrid.py`)
- Temporal action recognizer → DINO spatial encoder: encodes frames (evidence: `src/assembly/models/vit_lstm_recognizer.py`)
- Temporal action recognizer → BiLSTM classifier: classifies windows (evidence: `src/assembly/models/vit_lstm_recognizer.py`)
- Hybrid runtime → Slot and action fusion: fuses evidence (evidence: `scripts/run_hybrid.py`)
- Slot and action fusion → Earbud fusion logic: uses earbud rules (evidence: `src/assembly/slot_fusion.py`)
- Hybrid runtime → Prediction debouncer: stabilizes predictions (evidence: `scripts/run_hybrid.py`)
- Hybrid runtime → Assembly workflow FSM: submits action events (evidence: `scripts/run_hybrid.py`)
- Assembly workflow FSM → Workflow configuration: loads workflow rules (evidence: `src/assembly/fsm.py`)
- Hybrid runtime → Event monitor: updates event monitor (evidence: `scripts/run_hybrid.py`)
- Hybrid runtime → Runtime dashboard: updates display (evidence: `scripts/run_hybrid.py`)
- Runtime dashboard → Operator: shows status (evidence: `README.md`)
- Assembly workflow FSM → PASS, violation, next step: returns outcome (evidence: `src/assembly/fsm.py`)
- Action annotations → Action dataset: labels action data (evidence: `README.md`)
- Action dataset → Spatial features: provides video data (evidence: `README.md`)
- Spatial features → Action model training: supplies features (evidence: `README.md`)
- Action model training → Model evaluation: evaluates model (evidence: `README.md`)
