How hans-hc/violina works
Violina is a real-time, full-stack computer-vision posture-feedback system:
• Frontend (React/Vite) captures webcam video, renders UI, and displays posture feedback/overlays in real time.
• Backend (Flask + Python + MediaPipe) runs pose + hand landmark models, computes geometric posture metrics, compares against a baseline, and returns feedback.
• There is also optional hardware feedback (Arduino buzzer) bridged via a Node serial server.
• The repo includes offline/auxiliary pipelines: baseline capture, tuning, validation, and testing assets.
> Webcam → Frame streaming → ML inference → Posture evaluation → Feedback (UI overlays + possible buzzer) → Repeat
File tree to architectural “containers”
From the tree, you have clear container boundaries:
Frontend container (front-end/)
Key indicators:
• front-end/src/components/* includes Webcam.jsx, WebcamStream.jsx, PostureAnalysis.jsx, Tuner.jsx, Metronome.jsx.
• Vite entrypoints: front-end/src/main.jsx, front-end/src/App.jsx, front-end/vite.config.js.
What to represent:
• Webcam capture & streaming component(s)
• Posture feedback UI component(s)
• Tuning UI (likely adjusting thresholds / sensitivity)
• Session UX (Home page, Navbar, CSS)
ML / posture backend container (back-end/)
Key indicators:
• Flask server entrypoints mentioned in README: correctForm.py and webcam.py.
• ML assets: pose_landmarker.task, hand_landmarker.task (MediaPipe task models).
• Core logic modules: pose_comparator.py, drawingLines.py, possibly “geometry” logic in correctForm.py.
• Data & artifacts:
• baseline_images/, testing_images/
• output/*.json (baseline & results like baseline_data.json, validation_results.json)
What to represent:
• HTTP API that receives frames and returns feedback.
• Inference pipeline (pose + hand landmark detection).
• Posture evaluation (comparison to baseline, rule/metric computation).
• Overlay generation (lines/annotations).
• Baseline and tuning tools (offline scripts).
Serial / hardware feedback container (back-end/serial-server.js + Arduino sketch)
Key indicators:
• back-end/serial-server.js (Node process to talk to serial port)
• back-end/INCORRECT_BUZZER_SOUND/INCORRECT_BUZZER_SOUND.ino (Arduino)
What to represent:
• Optional alert channel: backend/frontend triggers → Node serial bridge → Arduino buzzer.
Repo-level Node artifacts (package.json, node_modules/)
• Build/dev servers (Vite dev server)
• Any top-level Node service if actually executed (unclear; the explicit runtime Node service is serial-server.js)
View 1: System Context (external actors + system)
Show:
• User (violin player)
• Webcam
• Violina System
• Optional external: Arduino device (buzzer)
Purpose: communicate who uses it and what peripherals exist.
This is the “main” architecture picture.
Containers:
• Browser (React/Vite app)
• Flask backend (Python)
• MediaPipe models (as a dependency inside backend)
• Baseline/Results storage (local filesystem JSON + images)
• Node Serial Server
• Arduino Buzzer
• Browser → Flask: “Frame stream” (likely HTTP POST or WebSocket)
• Flask → Browser: “Feedback JSON + overlay instructions (lines/landmarks)”
• Flask or Browser → Node Serial: “Incorrect posture event”
• Node Serial → Arduino: “Serial command”
• Backend ↔ Filesystem: “read/write baseline_data.json, validation_results.json, baseline_images”
Split the backend into components that match the files:
• API Layer (in correctForm.py and/or webcam.py)
• endpoints for: frame ingestion, baseline selection, tuning params, etc. (name them after you inspect actual route definitions)
• Inference Layer
• MediaPipe Pose Landmarker (pose_landmarker.task)
• MediaPipe Hand Landmarker (hand_landmarker.task)
• Posture Analysis Layer
• pose_comparator.py (baseline comparison)
• drawingLines.py (overlay geometry)
• any metric/rule computation in correctForm.py
• Baseline & Validation Tooling
• baseline_collector.py, run_baseline_collection.py
• validate_poses.py
• tuning.py, tuning_requirements.txt
• Artifacts
• output/*.json, baseline_images/, testing_images/
API → Inference → Analysis → Response builder → API response.
Produce
• posture status (ok / drifting / incorrect)
• suggestions
• overlay primitives (lines/points)
If incorrect: trigger buzzer event path (optional)
Given the README + tree, ensure these interactions are not vague:
Webcam and streaming boundary
• The webcam is accessed in the browser, but “streaming frames” crosses to the backend.
• This as a high-frequency call path; annotate “low latency, many requests/sec” or “WebSocket stream” (after code confirmation).
Baseline lifecycle
There’s a baseline pipeline indicated by:
• baseline_collector.py, run_baseline_collection.py, baseline_images/, output/baseline_data.json
• Baseline acquisition (offline or guided mode) → stored JSON/images → used by live comparator.
Model assets as deployable dependencies
The .task files are deployable artifacts and should be shown as:
• “Model files packaged with backend image/runtime”
Optional alerting hardware path
Keep it separate with a clear “optional” label:
• Posture incorrect event → serial server → Arduino buzzer
5) Diagramming guidelines tailored to this repo (be very detailed; split into many components)
Component naming: match the code
Use the actual file/module names to avoid abstraction drift:
• correctForm.py = “Flask API + posture evaluation orchestration”
• webcam.py = “webcam/frame source utility or streaming server” (confirm role in code)
• pose_comparator.py = “baseline comparison logic”
• drawingLines.py = “overlay primitives / landmark visualization”
• baseline_collector.py = “baseline capture pipeline”
• serial-server.js = “serial bridge service”
Distinguish runtime vs offline tools
In the backend, visually separate:
• Runtime services (Flask API, inference)
• Offline scripts (baseline collection, validation, tuning)
This set is detailed, matches the repo structure, and separates concerns without overthinking it—while still breaking the system into as many concrete components as possible.