How akirokazuki/bisindo works
1. Overall shape
- This repo is split into three main layers: a machine-learning pipeline (ml/), a real-time backend service (backend/), and a browser-based client (frontend/).
- The runtime product is a real-time BISINDO emergency sign detector; the training/evaluation code is separate from the live inference path.
2. Live inference data flow
- The user’s camera is accessed in the browser.
- MediaPipe Holistic runs on the client side to extract body/hand/face landmarks from each frame.
- Those landmarks are sent to the backend over WebSocket for low-latency streaming inference.
- The backend classifies the gesture using the trained ST-GCN model and returns detection results back to the UI.
3. Frontend subsystem
- frontend/ is a React + Vite application.
- App.jsx composes the main UI, while components like CameraView, DetectionDisplay, EmergencyContact, and SkeletonOverlay separate video capture, live status, contact actions, and visual skeleton rendering.
- Hooks such as useMediaPipe.js and useWebSocket.js isolate landmark extraction and socket communication from presentation logic.
4. Client-side pose extraction boundary
- The browser does more than render the UI: it performs feature extraction locally via MediaPipe.
- That means the backend does not receive raw video frames as the primary inference input; it receives skeletal landmark data, which reduces bandwidth and shifts computer-vision workload to the client.
- public/sw.js indicates support for service worker–based web push notifications.
5. Backend runtime
- backend/ is the real-time server, intended to run under FastAPI with a WebSocket endpoint.
- main.py is the service entrypoint; inference.py likely wraps model loading and prediction; buffer.py manages temporal frame/landmark buffering needed for sequence models; notification.py handles alerts after detection.
- The backend sits between the browser stream and external notification services.
6. Temporal buffering and sequence handling
- The model is sequence-based, so the backend must accumulate multiple frames/landmark samples before classifying.
- buffer.py is an important architectural piece because it bridges per-frame input from the frontend and the fixed-length temporal windows expected by ST-GCN.
- live_buffer_dump.npy suggests the project also captures live landmark sequences for debugging or offline inspection.
7. Notification subsystem
- Emergency detection triggers a multi-channel alert layer.
- The README mentions SMS via Textbee.dev, voice calls via Twilio, and web push notifications.
- notification.py is the backend boundary to those external services; tmp_twilio_test.py suggests Twilio integration was validated separately.
8. Machine-learning pipeline
- ml/ contains the offline pipeline for building the classifier.
- graph.py, model.py, and train.py indicate an ST-GCN-style architecture where skeleton joints are modeled as a graph and learned over time.
- This is the core model subsystem that turns landmark sequences into emergency-sign predictions.
9. Dataset preparation flow
- The training pipeline is staged: extract_keypoints.py extracts pose landmarks from raw videos, dataset.py organizes/splits the data, and augment.py expands the dataset.
- This indicates the training data is not directly raw video at model time; it is converted into structured keypoint sequences first.
- check_frames.py appears to be a utility for validating frame extraction or dataset quality.
10. Robustness and evaluation
- evaluate.py generates standard model-quality outputs such as confusion matrix and F1-score.
- evaluate_dropout.py specifically tests robustness to partial occlusion / keypoint dropout, which matches the project’s stated focus on resilience under missing landmarks.
- This is an important architectural concern because the live system depends on landmark completeness from MediaPipe.
11. Shared utilities and preprocessing
- utils/normalize.py suggests a shared normalization step used across training and inference to keep landmark representation consistent.
- This kind of utility is a key boundary: preprocessing must be identical between offline training and live inference or prediction quality will degrade.
12. Model artifact and deployment context
- holistic_landmarker.task is a MediaPipe asset used by the browser-side landmark extractor.
- requirements.txt at the repo root and another in bisindo-emergency/ indicate Python dependencies for backend/ML, while frontend/package.json defines the JavaScript runtime dependencies.
- Operationally, the system is deployed as two processes: a Python API server and a separate frontend dev/build server.
13. How to diagram it
- A correct architecture diagram should show:
- Browser camera input
- MediaPipe landmark extraction in the frontend
- WebSocket stream to FastAPI backend
- Buffering/sequence assembly
- ST-GCN inference
- Notification fan-out to SMS/voice/web push
- Separate offline ML pipeline feeding the trained model artifact used by inference
- The key boundary is that training is offline, while pose extraction and inference are split between browser and backend for real-time performance.
Main entry points: useMediaPipe.js MediaPipe · App.jsx App · main.py Main · inference.py Inference
Components 23
Frontend
React + Vite browser client with client-side landmark extraction
- App · Composes the live detection UI
bisindo-emergency/frontend/src/App.jsx - Camera View · Browser camera access and preview
bisindo-emergency/frontend/src/components/CameraView.jsx - MediaPipe · Extracts body, hand, and face landmarks in the browser
bisindo-emergency/frontend/src/hooks/useMediaPipe.js - WebSocket · Streams landmarks to the backend and receives results
bisindo-emergency/frontend/src/hooks/useWebSocket.js - Detection · Shows current classification and alerts
bisindo-emergency/frontend/src/components/DetectionDisplay.jsx - Overlay · Renders the tracked skeleton on top of the camera feed
bisindo-emergency/frontend/src/components/SkeletonOverlay.jsx - Contact · Emergency contact and notification actions in the UI
bisindo-emergency/frontend/src/components/EmergencyContact.jsx - Push SW · Supports web push notification handling in the browser
bisindo-emergency/frontend/public/sw.js
Backend
FastAPI real-time inference service and alert fan-out
- Main · FastAPI service startup and WebSocket endpoint wiring
bisindo-emergency/backend/main.py - Buffer · Accumulates landmark sequences for windowed inference
bisindo-emergency/backend/buffer.py - Inference · Loads the trained model and produces predictions
bisindo-emergency/backend/inference.py - Notify · Fans out emergency alerts to SMS, voice, and push services
bisindo-emergency/backend/notification.py
ML Pipeline
Offline training, evaluation, and dataset preparation for the ST-GCN model
- Train · Offline ST-GCN training entrypoint
bisindo-emergency/ml/train.py - Dataset · Organizes and splits keypoint datasets
bisindo-emergency/ml/dataset.py - Keypoints · Converts raw videos into landmark sequences
bisindo-emergency/ml/extract_keypoints.py - Augment · Expands the training set with transformed samples
bisindo-emergency/ml/augment.py - Graph Model · Defines the skeleton graph and temporal classifier
bisindo-emergency/ml/graph.py - Network · Learns gesture classes from landmark sequences
bisindo-emergency/ml/model.py - Evaluate · Runs standard evaluation and confusion-matrix analysis
bisindo-emergency/ml/evaluate.py - Dropout Eval · Tests performance under missing keypoints or occlusion
bisindo-emergency/ml/evaluate_dropout.py
Shared Assets
Assets and utilities reused across runtime and training
- Normalize · Keeps landmark preprocessing consistent across training and inference
bisindo-emergency/utils/normalize.py - Landmarker · Browser-side MediaPipe Holistic model asset
bisindo-emergency/holistic_landmarker.task - Buffer Dump · Captured live landmark sequence sample for inspection
bisindo-emergency/live_buffer_dump.npy
Connections 24
- App → Camera View: composes · App assembles the live capture view
- App → Detection: composes · App renders detection status
- App → Contact: composes · App exposes emergency actions
- Camera View → MediaPipe: frames · Captured camera frames are processed for landmarks
- MediaPipe → Landmarker: uses · Client-side extraction relies on the MediaPipe model asset
- MediaPipe → Overlay: feeds · Extracted landmarks drive the overlay visualization
- MediaPipe → WebSocket: streams · Landmark sequences are sent for inference
- WebSocket → Main: websocket · Browser connects to the backend inference socket
- Main → Buffer: routes · Incoming landmark samples are buffered into temporal windows
- Buffer → Inference: windows · Buffered sequences are passed to the classifier
- Inference → Normalize: preprocesses · Inference uses the same normalization as training
- Inference → Notify: triggers · Positive detections activate alert fan-out
- Main → Detection: returns · Classification results are sent back to the UI
- Notify → Push SW: push · Web push notifications are handled in the browser
- Notify → Contact: alerts · UI contact actions complement backend alerting
- Train → Dataset: uses · Training consumes prepared datasets
- Dataset → Keypoints: builds · Dataset preparation begins with keypoint extraction
- Dataset → Augment: expands · Augmentation increases dataset diversity
- Train → Graph Model: defines · Training instantiates the skeleton graph architecture
- Graph Model → Network: implements · Model definition encodes the ST-GCN network
- Evaluate → Network: scores · Evaluation runs the trained classifier on held-out data
- Dropout Eval → Network: stress tests · Robustness evaluation measures resilience to missing keypoints
- Keypoints → Normalize: aligns · Training and inference share preprocessing semantics
- Buffer Dump → Buffer: captures · Debug artifact records a live buffered sample