How muditgarg48/gesture_based_device_control works
Project type and purpose
• This is a full-stack Python application for real-time gesture recognition to control IoT devices. It combines a web interface (HTML/CSS/JS served by server.py), a back-end orchestrator (Flask or similar in server.py and scripts/), and a machine-learning pipeline (Mediapipe for key-point extraction, LSTM models in data/models, TensorFlow training, TensorBoard visualization).
File structure
• Top-level “server.py” plus “static/” and “templates/” imply a web server (MVC-style):
– templates/*.html are views
– static/js and static/css are assets
– server.py is the controller for HTTP endpoints
• “scripts/” contains modular business logic: camera_feed_testing, dataset_functions, mediapipe_functions, model_functions, control_firestick, etc.
• “data/” holds persisted artifacts: available_gestures.npy, scenarios folder for raw gesture captures, models folder with .h5 files.
• project_setup.py and project_integrity_check.py are pre-deployment tooling.
• main.ipynb is a demonstration notebook.
README for insights
• Confirms use of Mediapipe, OpenCV, NumPy, TensorFlow/Keras, and a virtual environment–based deploy.
• Clarifies user flows:
(a) Define gestures → (b) Capture video frames → (c) Preprocess key points → (d) Train LSTM → (e) Deploy inference → (f) Send commands to IoT device.
A. Front-end UI
– Browser running index.html + JS/CSS
– Sections: create scenario, view scenarios, global vars, initial checks
B. Web Server (server.py)
– HTTP routes for UI, triggers backend actions
– Invokes functions in scripts/
C. Gesture Pipeline
1. Camera Feed Module (scripts/camera_feed_testing.py)
2. Mediapipe Key-Point Extractor (scripts/mediapipe_functions.py)
3. Dataset Manager (scripts/dataset_functions.py; scenarios folder)
4. Gesture List Manager (scripts/gestures_functions.py; available_gestures.npy)
D. ML Training & Inference
– Model Trainer & Tester (scripts/model_functions.py)
– TensorBoard Monitor (scripts/tensorboard_training_monitor.py; tensorboard_logs/)
– Persisted Models (data/models/*.h5)
E. IoT Control Module
– Control Firestick or other devices (scripts/control_firestick.py)
– Communicates over local network
F. Data Store
– File-system-based storage for raw numpy arrays, models, scenarios
G. Pre-deployment Tools
– project_setup.py, project_integrity_check.py for bootstrapping environment
Relationships & Data Flows
• User → Browser sends HTTP requests to → Server
• Server → calls scripts/… to perform tasks
• Camera module → streams frames → Mediapipe → outputs key-point arrays → saved to scenarios/ or fed to model trainer
• Model trainer → reads from data/training_action_data & available_gestures.npy → trains LSTM → writes .h5 model to data/models
• Inference endpoint in server.py → loads model → accepts live key-point streams → predicts gesture → passes command to control_firestick → IoT device
• TensorBoard Monitor reads tensorboard_logs to display metrics
Architectural patterns & design principles
• MVC pattern on web stack: templates (View), server.py + JS (Controller), scripts as Model/Business Logic
• Modular separation: each functional area (data ingest, preprocessing, model, IoT control) lives in its own script
• File-system as lightweight database
Technologies, frameworks, libraries
• Backend: Python 3.11, Flask (or built-in server), Jinja2 templates
• ML: Mediapipe, TensorFlow/Keras, NumPy, OpenCV
• Front-end: HTML, CSS, vanilla JS
• Deployment & tooling: venv, pip, project_setup.py, project_integrity_check.py
• Monitoring: TensorBoard