How 23120340/region_based_convolutional_neural_network_demo works
Earbud Hybrid Assembly Monitor is a camera-based prototype that checks the sequence open case → insert first earbud → insert second earbud → close case and reports PASS, violation, or the next step. The main runtime combines object geometry from YOLO with temporal action recognition, fuses both signals, then checks workflow order. The README also describes a separate action-model data and training workflow. The graph reflects documented behavior and indexed source references; some internal wiring is not visible in the excerpts, so uncertain connections are omitted.
Main entry points: run_hybrid.py Hybrid runtime · vit_lstm_recognizer.py Temporal action recognizer · fsm.py Assembly workflow FSM · phone_camera.py Phone stream server
Components 25
Capture and UI
- Hybrid runtime
scripts/run_hybrid.py - Frame input
src/assembly/camera_input.py - Phone stream server
src/assembly/phone_camera.py - Phone connection UI
src/assembly/phone_camera_ui.py - Runtime dashboard
src/assembly/hybrid_dashboard.py
Perception
- YOLO detector
src/assembly/yolo_world_detector.py - Scene geometry
src/assembly/vision.py - Temporal action recognizer
src/assembly/models/vit_lstm_recognizer.py - DINO spatial encoder
src/assembly/models/spatial_encoder.py - BiLSTM classifier
src/assembly/models/action_net.py
Fusion and workflow
- Slot and action fusion
src/assembly/slot_fusion.py - Earbud fusion logic
src/assembly/earbud_fusion.py - Prediction debouncer
src/assembly/smoother.py - Assembly workflow FSM
src/assembly/fsm.py - Workflow configuration
src/assembly/config.py - Event monitor
src/assembly/monitor.py
Action model development
- Action annotations
scripts/annotate_actions.py - Action dataset
src/assembly/action_dataset.py - Spatial features
scripts/extract_spatial_features.py - Action model training
scripts/train_action_model.py - Model evaluation
scripts/evaluate_action_model.py
Other components
- Operator
- Camera or video
- Phone camera
- PASS, violation, next step
Connections 25
- Operator → Hybrid runtime: starts monitor
README.md - Camera or video → Frame input: supplies frames
README.md - Phone camera → Phone stream server: streams video
README.md - Hybrid runtime → Frame input: reads frames
scripts/run_hybrid.py - Hybrid runtime → Phone stream server: starts optional stream
scripts/run_hybrid.py - Hybrid runtime → Phone connection UI: opens connection UI
scripts/run_hybrid.py - Phone connection UI → Phone stream server: connects session
src/assembly/phone_camera_ui.py - Hybrid runtime → YOLO detector: runs detection
scripts/run_hybrid.py - YOLO detector → Scene geometry: produces detections
src/assembly/yolo_world_detector.py - Hybrid runtime → Temporal action recognizer: predicts actions
scripts/run_hybrid.py - Temporal action recognizer → DINO spatial encoder: encodes frames
src/assembly/models/vit_lstm_recognizer.py - Temporal action recognizer → BiLSTM classifier: classifies windows
src/assembly/models/vit_lstm_recognizer.py - Hybrid runtime → Slot and action fusion: fuses evidence
scripts/run_hybrid.py - Slot and action fusion → Earbud fusion logic: uses earbud rules
src/assembly/slot_fusion.py - Hybrid runtime → Prediction debouncer: stabilizes predictions
scripts/run_hybrid.py - Hybrid runtime → Assembly workflow FSM: submits action events
scripts/run_hybrid.py - Assembly workflow FSM → Workflow configuration: loads workflow rules
src/assembly/fsm.py - Hybrid runtime → Event monitor: updates event monitor
scripts/run_hybrid.py - Hybrid runtime → Runtime dashboard: updates display
scripts/run_hybrid.py - Runtime dashboard → Operator: shows status
README.md - Assembly workflow FSM → PASS, violation, next step: returns outcome
src/assembly/fsm.py - Action annotations → Action dataset: labels action data
README.md - Action dataset → Spatial features: provides video data
README.md - Spatial features → Action model training: supplies features
README.md - Action model training → Model evaluation: evaluates model
README.md