# akirokazuki/bisindo architecture

> Architecture diagram of the GitHub repository akirokazuki/bisindo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/akirokazuki/bisindo
- Repository: https://github.com/akirokazuki/bisindo

## Overview

1. **Overall shape**
   - This repo is split into three main layers: a **machine-learning pipeline** (`ml/`), a **real-time backend service** (`backend/`), and a **browser-based client** (`frontend/`).
   - The runtime product is a **real-time BISINDO emergency sign detector**; the training/evaluation code is separate from the live inference path.

2. **Live inference data flow**
   - The user’s camera is accessed in the browser.
   - **MediaPipe Holistic** runs on the client side to extract body/hand/face landmarks from each frame.
   - Those landmarks are sent to the backend over **WebSocket** for low-latency streaming inference.
   - The backend classifies the gesture using the trained ST-GCN model and returns detection results back to the UI.

3. **Frontend subsystem**
   - `frontend/` is a **React + Vite** application.
   - `App.jsx` composes the main UI, while components like `CameraView`, `DetectionDisplay`, `EmergencyContact`, and `SkeletonOverlay` separate video capture, live status, contact actions, and visual skeleton rendering.
   - Hooks such as `useMediaPipe.js` and `useWebSocket.js` isolate landmark extraction and socket communication from presentation logic.

4. **Client-side pose extraction boundary**
   - The browser does more than render the UI: it performs **feature extraction** locally via MediaPipe.
   - That means the backend does not receive raw video frames as the primary inference input; it receives **skeletal landmark data**, which reduces bandwidth and shifts computer-vision workload to the client.
   - `public/sw.js` indicates support for **service worker–based web push notifications**.

5. **Backend runtime**
   - `backend/` is the real-time server, intended to run under **FastAPI** with a WebSocket endpoint.
   - `main.py` is the service entrypoint; `inference.py` likely wraps model loading and prediction; `buffer.py` manages temporal frame/landmark buffering needed for sequence models; `notification.py` handles alerts after detection.
   - The backend sits between the browser stream and external notification services.

6. **Temporal buffering and sequence handling**
   - The model is sequence-based, so the backend must accumulate multiple frames/landmark samples before classifying.
   - `buffer.py` is an important architectural piece because it bridges per-frame input from the frontend and the fixed-length temporal windows expected by ST-GCN.
   - `live_buffer_dump.npy` suggests the project also captures live landmark sequences for debugging or offline inspection.

7. **Notification subsystem**
   - Emergency detection triggers a **multi-channel alert layer**.
   - The README mentions **SMS via Textbee.dev**, **voice calls via Twilio**, and **web push notifications**.
   - `notification.py` is the backend boundary to those external services; `tmp_twilio_test.py` suggests Twilio integration was validated separately.

8. **Machine-learning pipeline**
   - `ml/` contains the offline pipeline for building the classifier.
   - `graph.py`, `model.py`, and `train.py` indicate an ST-GCN-style architecture where skeleton joints are modeled as a graph and learned over time.
   - This is the core model subsystem that turns landmark sequences into emergency-sign predictions.

9. **Dataset preparation flow**
   - The training pipeline is staged: `extract_keypoints.py` extracts pose landmarks from raw videos, `dataset.py` organizes/splits the data, and `augment.py` expands the dataset.
   - This indicates the training data is not directly raw video at model time; it is converted into **structured keypoint sequences** first.
   - `check_frames.py` appears to be a utility for validating frame extraction or dataset quality.

10. **Robustness and evaluation**
   - `evaluate.py` generates standard model-quality outputs such as confusion matrix and F1-score.
   - `evaluate_dropout.py` specifically tests robustness to **partial occlusion / keypoint dropout**, which matches the project’s stated focus on resilience under missing landmarks.
   - This is an important architectural concern because the live system depends on landmark completeness from MediaPipe.

11. **Shared utilities and preprocessing**
   - `utils/normalize.py` suggests a shared normalization step used across training and inference to keep landmark representation consistent.
   - This kind of utility is a key boundary: preprocessing must be identical between offline training and live inference or prediction quality will degrade.

12. **Model artifact and deployment context**
   - `holistic_landmarker.task` is a MediaPipe asset used by the browser-side landmark extractor.
   - `requirements.txt` at the repo root and another in `bisindo-emergency/` indicate Python dependencies for backend/ML, while `frontend/package.json` defines the JavaScript runtime dependencies.
   - Operationally, the system is deployed as two processes: a Python API server and a separate frontend dev/build server.

13. **How to diagram it**
   - A correct architecture diagram should show:
     - Browser camera input
     - MediaPipe landmark extraction in the frontend
     - WebSocket stream to FastAPI backend
     - Buffering/sequence assembly
     - ST-GCN inference
     - Notification fan-out to SMS/voice/web push
     - Separate offline ML pipeline feeding the trained model artifact used by inference
   - The key boundary is that **training is offline**, while **pose extraction and inference are split between browser and backend** for real-time performance.

Main entry points: [`bisindo-emergency/frontend/src/hooks/useMediaPipe.js`](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/frontend/src/hooks/useMediaPipe.js) (MediaPipe), [`bisindo-emergency/frontend/src/App.jsx`](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/frontend/src/App.jsx) (App), [`bisindo-emergency/backend/main.py`](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/backend/main.py) (Main), [`bisindo-emergency/backend/inference.py`](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/backend/inference.py) (Inference)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_frontend["Frontend"]
  node_frontend_app["App<br/>react shell<br/>[App.jsx]"]
  node_camera_view["Camera View<br/>video capture<br/>[CameraView.jsx]"]
  node_mediapipe_hook["MediaPipe<br/>landmark hook<br/>[useMediaPipe.js]"]
  node_ws_hook["WebSocket<br/>socket hook<br/>[useWebSocket.js]"]
  node_detection_display["Detection<br/>status panel"]
  node_skeleton_overlay["Overlay<br/>visualizer"]
  node_emergency_contact["Contact<br/>action panel"]
  node_service_worker["Push SW<br/>service worker<br/>[sw.js]"]
end

subgraph group_backend["Backend"]
  node_backend_main["Main<br/>api entrypoint<br/>[main.py]"]
  node_buffer["Buffer<br/>[buffer.py]"]
  node_inference["Inference<br/>model wrapper<br/>[inference.py]"]
  node_notification["Notify<br/>alert gateway<br/>[notification.py]"]
end

subgraph group_ml["ML Pipeline"]
  node_train["Train<br/>[train.py]"]
  node_dataset["Dataset<br/>data prep<br/>[dataset.py]"]
  node_extract_keypoints["Keypoints<br/>feature extraction"]
  node_augment["Augment<br/>[augment.py]"]
  node_graph_model["Graph Model<br/>st-gcn model<br/>[graph.py]"]
  node_model["Network<br/>model definition<br/>[model.py]"]
  node_evaluate["Evaluate<br/>metrics<br/>[evaluate.py]"]
  node_evaluate_dropout["Dropout Eval<br/>robustness test"]
end

subgraph group_shared["Shared Assets"]
  node_normalize["Normalize<br/>shared preprocessing<br/>[normalize.py]"]
  node_landmarker_asset["Landmarker<br/>mediapipe asset"]
  node_buffer_dump["Buffer Dump<br/>debug artifact"]
end

node_frontend_app -->|"composes"| node_camera_view
node_frontend_app -->|"composes"| node_detection_display
node_frontend_app -->|"composes"| node_emergency_contact
node_camera_view -->|"frames"| node_mediapipe_hook
node_mediapipe_hook -->|"uses"| node_landmarker_asset
node_mediapipe_hook -->|"feeds"| node_skeleton_overlay
node_mediapipe_hook -->|"streams"| node_ws_hook
node_ws_hook -->|"websocket"| node_backend_main
node_backend_main -->|"routes"| node_buffer
node_buffer -->|"windows"| node_inference
node_inference -->|"preprocesses"| node_normalize
node_inference -->|"triggers"| node_notification
node_backend_main -->|"returns"| node_detection_display
node_notification -->|"push"| node_service_worker
node_notification -->|"alerts"| node_emergency_contact
node_train -->|"uses"| node_dataset
node_dataset -->|"builds"| node_extract_keypoints
node_dataset -->|"expands"| node_augment
node_train -->|"defines"| node_graph_model
node_graph_model -->|"implements"| node_model
node_evaluate -->|"scores"| node_model
node_evaluate_dropout -->|"stress tests"| node_model
node_extract_keypoints -->|"aligns"| node_normalize
node_buffer_dump -->|"captures"| node_buffer

click node_frontend_app "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/frontend/src/App.jsx"
click node_camera_view "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/frontend/src/components/CameraView.jsx"
click node_mediapipe_hook "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/frontend/src/hooks/useMediaPipe.js"
click node_ws_hook "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/frontend/src/hooks/useWebSocket.js"
click node_detection_display "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/frontend/src/components/DetectionDisplay.jsx"
click node_skeleton_overlay "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/frontend/src/components/SkeletonOverlay.jsx"
click node_emergency_contact "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/frontend/src/components/EmergencyContact.jsx"
click node_service_worker "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/frontend/public/sw.js"
click node_backend_main "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/backend/main.py"
click node_buffer "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/backend/buffer.py"
click node_inference "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/backend/inference.py"
click node_notification "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/backend/notification.py"
click node_train "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/ml/train.py"
click node_dataset "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/ml/dataset.py"
click node_extract_keypoints "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/ml/extract_keypoints.py"
click node_augment "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/ml/augment.py"
click node_graph_model "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/ml/graph.py"
click node_model "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/ml/model.py"
click node_evaluate "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/ml/evaluate.py"
click node_evaluate_dropout "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/ml/evaluate_dropout.py"
click node_normalize "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/utils/normalize.py"
click node_landmarker_asset "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/holistic_landmarker.task"
click node_buffer_dump "https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/live_buffer_dump.npy"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_frontend_app,node_camera_view,node_mediapipe_hook,node_ws_hook,node_detection_display,node_skeleton_overlay,node_emergency_contact,node_service_worker toneBlue
class node_backend_main,node_buffer,node_inference,node_notification toneAmber
class node_train,node_dataset,node_extract_keypoints,node_augment,node_graph_model,node_model,node_evaluate,node_evaluate_dropout toneMint
class node_normalize,node_landmarker_asset,node_buffer_dump toneRose
```

## Components

### Frontend

React + Vite browser client with client-side landmark extraction

- [App](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/frontend/src/App.jsx): react shell · Composes the live detection UI · `bisindo-emergency/frontend/src/App.jsx`
- [Camera View](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/frontend/src/components/CameraView.jsx): video capture · Browser camera access and preview · `bisindo-emergency/frontend/src/components/CameraView.jsx`
- [MediaPipe](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/frontend/src/hooks/useMediaPipe.js): landmark hook · Extracts body, hand, and face landmarks in the browser · `bisindo-emergency/frontend/src/hooks/useMediaPipe.js`
- [WebSocket](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/frontend/src/hooks/useWebSocket.js): socket hook · Streams landmarks to the backend and receives results · `bisindo-emergency/frontend/src/hooks/useWebSocket.js`
- [Detection](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/frontend/src/components/DetectionDisplay.jsx): status panel · Shows current classification and alerts · `bisindo-emergency/frontend/src/components/DetectionDisplay.jsx`
- [Overlay](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/frontend/src/components/SkeletonOverlay.jsx): visualizer · Renders the tracked skeleton on top of the camera feed · `bisindo-emergency/frontend/src/components/SkeletonOverlay.jsx`
- [Contact](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/frontend/src/components/EmergencyContact.jsx): action panel · Emergency contact and notification actions in the UI · `bisindo-emergency/frontend/src/components/EmergencyContact.jsx`
- [Push SW](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/frontend/public/sw.js): service worker · Supports web push notification handling in the browser · `bisindo-emergency/frontend/public/sw.js`

### Backend

FastAPI real-time inference service and alert fan-out

- [Main](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/backend/main.py): api entrypoint · FastAPI service startup and WebSocket endpoint wiring · `bisindo-emergency/backend/main.py`
- [Buffer](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/backend/buffer.py): temporal buffer · Accumulates landmark sequences for windowed inference · `bisindo-emergency/backend/buffer.py`
- [Inference](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/backend/inference.py): model wrapper · Loads the trained model and produces predictions · `bisindo-emergency/backend/inference.py`
- [Notify](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/backend/notification.py): alert gateway · Fans out emergency alerts to SMS, voice, and push services · `bisindo-emergency/backend/notification.py`

### ML Pipeline

Offline training, evaluation, and dataset preparation for the ST-GCN model

- [Train](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/ml/train.py): training pipeline · Offline ST-GCN training entrypoint · `bisindo-emergency/ml/train.py`
- [Dataset](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/ml/dataset.py): data prep · Organizes and splits keypoint datasets · `bisindo-emergency/ml/dataset.py`
- [Keypoints](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/ml/extract_keypoints.py): feature extraction · Converts raw videos into landmark sequences · `bisindo-emergency/ml/extract_keypoints.py`
- [Augment](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/ml/augment.py): data augmentation · Expands the training set with transformed samples · `bisindo-emergency/ml/augment.py`
- [Graph Model](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/ml/graph.py): st-gcn model · Defines the skeleton graph and temporal classifier · `bisindo-emergency/ml/graph.py`
- [Network](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/ml/model.py): model definition · Learns gesture classes from landmark sequences · `bisindo-emergency/ml/model.py`
- [Evaluate](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/ml/evaluate.py): metrics · Runs standard evaluation and confusion-matrix analysis · `bisindo-emergency/ml/evaluate.py`
- [Dropout Eval](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/ml/evaluate_dropout.py): robustness test · Tests performance under missing keypoints or occlusion · `bisindo-emergency/ml/evaluate_dropout.py`

### Shared Assets

Assets and utilities reused across runtime and training

- [Normalize](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/utils/normalize.py): shared preprocessing · Keeps landmark preprocessing consistent across training and inference · `bisindo-emergency/utils/normalize.py`
- [Landmarker](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/holistic_landmarker.task): mediapipe asset · Browser-side MediaPipe Holistic model asset · `bisindo-emergency/holistic_landmarker.task`
- [Buffer Dump](https://github.com/akirokazuki/bisindo/blob/main/bisindo-emergency/live_buffer_dump.npy): debug artifact · Captured live landmark sequence sample for inspection · `bisindo-emergency/live_buffer_dump.npy`

## Connections

- App → Camera View: composes · App assembles the live capture view
- App → Detection: composes · App renders detection status
- App → Contact: composes · App exposes emergency actions
- Camera View → MediaPipe: frames · Captured camera frames are processed for landmarks
- MediaPipe → Landmarker: uses · Client-side extraction relies on the MediaPipe model asset
- MediaPipe → Overlay: feeds · Extracted landmarks drive the overlay visualization
- MediaPipe → WebSocket: streams · Landmark sequences are sent for inference
- WebSocket → Main: websocket · Browser connects to the backend inference socket
- Main → Buffer: routes · Incoming landmark samples are buffered into temporal windows
- Buffer → Inference: windows · Buffered sequences are passed to the classifier
- Inference → Normalize: preprocesses · Inference uses the same normalization as training
- Inference → Notify: triggers · Positive detections activate alert fan-out
- Main → Detection: returns · Classification results are sent back to the UI
- Notify → Push SW: push · Web push notifications are handled in the browser
- Notify → Contact: alerts · UI contact actions complement backend alerting
- Train → Dataset: uses · Training consumes prepared datasets
- Dataset → Keypoints: builds · Dataset preparation begins with keypoint extraction
- Dataset → Augment: expands · Augmentation increases dataset diversity
- Train → Graph Model: defines · Training instantiates the skeleton graph architecture
- Graph Model → Network: implements · Model definition encodes the ST-GCN network
- Evaluate → Network: scores · Evaluation runs the trained classifier on held-out data
- Dropout Eval → Network: stress tests · Robustness evaluation measures resilience to missing keypoints
- Keypoints → Normalize: aligns · Training and inference share preprocessing semantics
- Buffer Dump → Buffer: captures · Debug artifact records a live buffered sample
