How viral-doshi/gesture-controlled-virtual-mouse works
Main Components
• Gesture Recognition Module – This includes files like Gesture_Controller.py and Gesture_Controller_Gloved.py that implement hand tracking and gesture-based cursor control. There is a specific branch for “gloved” operation versus naked-hand detection using MediaPipe.
• Voice Assistant Module – Represented by Proton.py, this module manages voice commands to perform tasks from launching/stopping gesture recognition to performing actions like Google search, file navigation, and controlling device volume/brightness.
• Web Interface Component – The src/web folder contains HTML, CSS, and JavaScript files. This suggests that the project includes a simple front-end, potentially for visual feedback or status display.
• Supporting resources – Calibration images and configuration files found in src/calib_images and the demo_media (videos and gifs) which illustrate the gestures and outcomes.
• External Libraries/Frameworks – The README and project metadata indicate a dependency on Python 3.8.5, MediaPipe, pybind11 (for the underlying models), PyAudio, and pywin32. These should be highlighted as they are critical to the system’s operation.
Relationships and Interactions
• The Gesture Recognition module captures frames from the webcam and processes them using computer vision algorithms to detect hand gestures. These gestures are then mapped to mouse actions (cursor movement, clicking, scrolling, etc.).
• The Voice Assistant module operates alongside the gesture controller. It listens for voice commands (using PyAudio and related libraries) and triggers corresponding actions in the system. It may also control the start and stop of the gesture recognition thread.
• In certain cases, the two modules may interact; for example, a voice command (“Proton Launch Gesture Recognition”) causes the gesture recognition to initiate.
• The web component (found under src/web) may provide a UI layer to display status, instructions, or visual feedback of the gesture recognition process, effectively serving as an additional presentation layer.
Architectural Patterns and Principles
• Modular Design – The system is clearly divided between gesture recognition and voice control as separate modules. Each module has its dedicated entry scripts and resources.
• Separation of Concerns – Gesture functionality and voice command functionality are decoupled from each other, which enables isolated development and testing.