How pramzie/air-piano works
Project Type and Purpose
• This project is a specialized computer vision and MIDI synthesis application. Its primary function is to detect hand gestures using a webcam and computer vision libraries (OpenCV and cvzone) and translate these gestures into MIDI signals using pygame.midi, specifically to play chords in the D scale.
• The README clearly states it operates as a hand-tracking MIDI controller with automatic sustain for notes, making it both an interactive musical instrument and a demonstration of gesture-controlled interfaces.
Analyzing the File Structure
• Chord_mapping.txt: Likely contains the mapping between different hand/finger gestures and the corresponding D scale chords.
• hand_dscale.py: The main Python script coordinating the hand tracking, gesture recognition, and MIDI output.
• README.md: Contains the project description, features, and high-level overview.
• requirements01.txt: Lists the dependencies not only to set up the development environment but also indicates the external libraries used (such as OpenCV, cvzone, and pygame.midi).
Main Components
a. Input/Acquisition Layer:
• Webcam/Capture Module: Represents the video input device from which frames are captured.
b. Vision Processing & Hand Tracking Module:
• OpenCV and cvzone integration that processes the video stream to detect hand gestures.
• This module represents the computer vision pipeline to extract finger positions and gestures.
c. Gesture-to-Chord Mapping:
• Chord Mapping Component: A logical module that loads the chord mappings from Chord_mapping.txt and maps detected gestures to the correct D scale chords.
d. MIDI Output Module:
• Pygame MIDI Module: Handles the generation and transmission of MIDI signals to simulate piano sounds. This module also implements the automatic sustain for 2 seconds.
e. Control & Orchestration:
• Main Application Logic within hand_dscale.py that ties all the above modules together, managing the capture of video frames, gesture detection, chord mapping, and MIDI signal output.
f. Dependency/Configuration Management:
• Dependencies listed in requirements01.txt, indicating the external libraries used.
Relationships and Data Flow
• Input flows from the Webcam to the Vision Processing module.
• The Vision Processing module outputs recognized hand or finger positions.
• These positions are sent to the Gesture-to-Chord Mapping module which decides which chord (from the D scale) is indicated.
• The decision is then passed to the MIDI Output Module, which uses pygame.midi to send the MIDI signals.
• A feedback loop to account for the automatic sustain (notes continue for a fixed duration and then stop).
Important Architectural Patterns and Technologies
• Modular Design: Each module (input, processing, mapping, and MIDI output) is logically separated, which helps in understanding dependencies and future extensibility.
• Event Flow: Gestures trigger events leading to MIDI outputs.
• Technologies included: Python, OpenCV, cvzone, pygame.midi.
• The system might not be highly layered due to its simplicity; instead, it uses a linear data processing pipeline from capture to output.