# muditgarg48/gesture_based_device_control architecture

> Architecture diagram of the GitHub repository muditgarg48/gesture_based_device_control, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/muditgarg48/gesture_based_device_control
- Repository: https://github.com/muditgarg48/gesture_based_device_control

## Overview

##### Project type and purpose
- This is a full-stack Python application for real-time gesture recognition to control IoT devices. It combines a web interface (HTML/CSS/JS served by server.py), a back-end orchestrator (Flask or similar in server.py and scripts/), and a machine-learning pipeline (Mediapipe for key-point extraction, LSTM models in data/models, TensorFlow training, TensorBoard visualization).

##### File structure
- Top-level “server.py” plus “static/” and “templates/” imply a web server (MVC-style):
– templates/*.html are views
– static/js and static/css are assets
– server.py is the controller for HTTP endpoints
- “scripts/” contains modular business logic: camera_feed_testing, dataset_functions, mediapipe_functions, model_functions, control_firestick, etc.
- “data/” holds persisted artifacts: available_gestures.npy, scenarios folder for raw gesture captures, models folder with .h5 files.
- project_setup.py and project_integrity_check.py are pre-deployment tooling.
- main.ipynb is a demonstration notebook.

##### README for insights
- Confirms use of Mediapipe, OpenCV, NumPy, TensorFlow/Keras, and a virtual environment–based deploy.
- Clarifies user flows:
(a) Define gestures → (b) Capture video frames → (c) Preprocess key points → (d) Train LSTM → (e) Deploy inference → (f) Send commands to IoT device.

A. Front-end UI
– Browser running index.html + JS/CSS
– Sections: create scenario, view scenarios, global vars, initial checks
B. Web Server (server.py)
– HTTP routes for UI, triggers backend actions
– Invokes functions in scripts/
C. Gesture Pipeline
1. Camera Feed Module (scripts/camera_feed_testing.py)
2. Mediapipe Key-Point Extractor (scripts/mediapipe_functions.py)
3. Dataset Manager (scripts/dataset_functions.py; scenarios folder)
4. Gesture List Manager (scripts/gestures_functions.py; available_gestures.npy)
D. ML Training & Inference
– Model Trainer & Tester (scripts/model_functions.py)
– TensorBoard Monitor (scripts/tensorboard_training_monitor.py; tensorboard_logs/)
– Persisted Models (data/models/*.h5)
E. IoT Control Module
– Control Firestick or other devices (scripts/control_firestick.py)
– Communicates over local network
F. Data Store
– File-system-based storage for raw numpy arrays, models, scenarios
G. Pre-deployment Tools
– project_setup.py, project_integrity_check.py for bootstrapping environment

##### Relationships & Data Flows
- User → Browser sends HTTP requests to → Server
- Server → calls scripts/… to perform tasks
- Camera module → streams frames → Mediapipe → outputs key-point arrays → saved to scenarios/ or fed to model trainer
- Model trainer → reads from data/training_action_data & available_gestures.npy → trains LSTM → writes .h5 model to data/models
- Inference endpoint in server.py → loads model → accepts live key-point streams → predicts gesture → passes command to control_firestick → IoT device
- TensorBoard Monitor reads tensorboard_logs to display metrics

##### Architectural patterns & design principles
- MVC pattern on web stack: templates (View), server.py + JS (Controller), scripts as Model/Business Logic
- Modular separation: each functional area (data ingest, preprocessing, model, IoT control) lives in its own script
- File-system as lightweight database

##### Technologies, frameworks, libraries
- Backend: Python 3.11, Flask (or built-in server), Jinja2 templates
- ML: Mediapipe, TensorFlow/Keras, NumPy, OpenCV
- Front-end: HTML, CSS, vanilla JS
- Deployment & tooling: venv, pip, project_setup.py, project_integrity_check.py
- Monitoring: TensorBoard

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Front-end UI"
        Browser["Browser"]:::frontend
        IndexHTML["index.html"]:::frontend
        IndexJS["index.js"]:::frontend
        CSS1["collapsing.css"]:::frontend
        CSS2["index.css"]:::frontend
        CSS3["section_create_scenario.css"]:::frontend
        CSS4["section_global_vars.css"]:::frontend
        CSS5["section_initial_checks.css"]:::frontend
        CSS6["section_view_scenarios.css"]:::frontend
    end

    subgraph "Web Server (Controller)"
        Server["server.py"]:::backend
    end

    subgraph "Business Logic & ML Pipeline"
        CF["Camera Feed\nscripts/camera_feed_testing.py"]:::backend
        MP["Mediapipe Extractor\nscripts/mediapipe_functions.py"]:::ml
        DS["Dataset Manager\nscripts/dataset_functions.py"]:::backend
        GL["Gesture List Manager\nscripts/gestures_functions.py"]:::backend
        MF["Model Trainer & Tester\nscripts/model_functions.py"]:::ml
        TB["TensorBoard Monitor\nscripts/tensorboard_training_monitor.py"]:::external
        CM["IoT Control Module\nscripts/control_firestick.py"]:::external
        GV1["Fixed Vars\nscripts/global_variables/fixed.py"]:::backend
        GV2["User Vars\nscripts/global_variables/user_specific.py"]:::backend
        CL["CLI Functions\nscripts/commandline_functions.py"]:::backend
        MS["Main Script\nscripts/main_script.py"]:::backend
    end

    subgraph "Data Storage"
        Scenarios["scenarios/"]:::storage
        GesturesFile["available_gestures.npy"]:::storage
        Model1["final_test.h5"]:::storage
        Model2["test1.h5"]:::storage
        Model3["test2.h5"]:::storage
        Logs["tensorboard_logs/"]:::storage
    end

    subgraph "Pre-deployment Tools"
        Setup["project_setup.py"]:::tools
        Integrity["project_integrity_check.py"]:::tools
    end

    subgraph "External Systems"
        IoT["IoT Device"]:::external
    end

    Browser -->|HTTP GET| IndexHTML
    IndexHTML -->|includes| IndexJS
    IndexHTML --> CSS1
    IndexHTML --> CSS2
    IndexHTML --> CSS3
    IndexHTML --> CSS4
    IndexHTML --> CSS5
    IndexHTML --> CSS6

    Browser -->|interacts| Server
    Server -->|calls| CF
    Server -->|calls| MP
    Server -->|calls| DS
    Server -->|calls| GL
    Server -->|calls| MF
    Server -->|calls| CM
    CF -->|streams frames| MP
    MP -->|outputs keypoints| DS
    DS --> Scenarios
    GL --> GesturesFile
    MF -->|reads| Scenarios
    MF -->|reads| GesturesFile
    MF -->|writes| Model1
    MF -->|writes| Model2
    MF -->|writes| Model3
    MF -->|logs to| Logs
    Server -->|loads model| Model1
    Server -->|inference| MP
    Server -->|sends command| CM
    CM -->|network| IoT
    TB -->|monitors| Logs

    Setup -->|sets up| Server
    Integrity -->|verifies| Server

    click IndexHTML "https://github.com/muditgarg48/gesture_based_device_control/blob/main/templates/index.html"
    click IndexJS "https://github.com/muditgarg48/gesture_based_device_control/blob/main/static/js/index.js"
    click CSS1 "https://github.com/muditgarg48/gesture_based_device_control/blob/main/static/css/collapsing.css"
    click CSS2 "https://github.com/muditgarg48/gesture_based_device_control/blob/main/static/css/index.css"
    click CSS3 "https://github.com/muditgarg48/gesture_based_device_control/blob/main/static/css/section_create_scenario.css"
    click CSS4 "https://github.com/muditgarg48/gesture_based_device_control/blob/main/static/css/section_global_vars.css"
    click CSS5 "https://github.com/muditgarg48/gesture_based_device_control/blob/main/static/css/section_initial_checks.css"
    click CSS6 "https://github.com/muditgarg48/gesture_based_device_control/blob/main/static/css/section_view_scenarios.css"
    click Server "https://github.com/muditgarg48/gesture_based_device_control/blob/main/server.py"
    click CF "https://github.com/muditgarg48/gesture_based_device_control/blob/main/scripts/camera_feed_testing.py"
    click MP "https://github.com/muditgarg48/gesture_based_device_control/blob/main/scripts/mediapipe_functions.py"
    click DS "https://github.com/muditgarg48/gesture_based_device_control/blob/main/scripts/dataset_functions.py"
    click GL "https://github.com/muditgarg48/gesture_based_device_control/blob/main/scripts/gestures_functions.py"
    click MF "https://github.com/muditgarg48/gesture_based_device_control/blob/main/scripts/model_functions.py"
    click TB "https://github.com/muditgarg48/gesture_based_device_control/blob/main/scripts/tensorboard_training_monitor.py"
    click CM "https://github.com/muditgarg48/gesture_based_device_control/blob/main/scripts/control_firestick.py"
    click Scenarios "https://github.com/muditgarg48/gesture_based_device_control/tree/main/scenarios/"
    click GesturesFile "https://github.com/muditgarg48/gesture_based_device_control/blob/main/data/available_gestures.npy"
    click Model1 "https://github.com/muditgarg48/gesture_based_device_control/blob/main/data/models/final_test.h5"
    click Model2 "https://github.com/muditgarg48/gesture_based_device_control/blob/main/data/models/test1.h5"
    click Model3 "https://github.com/muditgarg48/gesture_based_device_control/blob/main/data/models/test2.h5"
    click CL "https://github.com/muditgarg48/gesture_based_device_control/blob/main/scripts/commandline_functions.py"
    click MS "https://github.com/muditgarg48/gesture_based_device_control/blob/main/scripts/main_script.py"
    click Setup "https://github.com/muditgarg48/gesture_based_device_control/blob/main/project_setup.py"
    click Integrity "https://github.com/muditgarg48/gesture_based_device_control/blob/main/project_integrity_check.py"

    classDef frontend fill:#D0E8FF,stroke:#333,stroke-width:1px
    classDef backend fill:#D0FFD0,stroke:#333,stroke-width:1px
    classDef ml fill:#FFE0B2,stroke:#333,stroke-width:1px
    classDef storage fill:#E0E0E0,stroke:#333,stroke-width:1px
    classDef tools fill:#F0F0F0,stroke:#333,stroke-width:1px
    classDef external fill:#E1BEE7,stroke:#333,stroke-width:1px
```
