How kazuhito00/hand-gesture-recognition-using-mediapipe works
Project Type and Purpose
• This is a standalone computer-vision/ML application (not a web or full-stack service).
• It uses MediaPipe (Python) to extract hand landmarks from a webcam feed, then classifies hand poses and finger‐motion gestures using two separate lightweight ML models (TFLite/HDF5).
• It also includes data‐collection modes and Jupyter notebooks for retraining those models.
File Structure
• Top-level:
• app.py → main demo & data‐collection driver
• keypoint_classification.ipynb, point_history_classification.ipynb
• requirements.txt, Dockerfile, .dockerignore
• model/
├─ keypoint_classifier/ → CSV dataset, inference wrapper, HDF5 & TFLite hand‐sign model, labels
└─ point_history_classifier/→ CSV dataset, inference wrapper, HDF5 & TFLite finger‐motion model, labels
• utils/
└─ cvfpscalc.py → FPS calculation helper
Core Components
a. Input Layer
• Webcam Device (OpenCV)
b. Preprocessing & Landmark Extraction
• MediaPipe Hands pipeline → outputs a stream of 21 keypoints
c. Inference Modules
1. KeypointClassifier
• Loads keypoint_classifier.tflite (or .hdf5)
• Preprocesses landmarks → MLP inference → hand‐sign label
2. PointHistoryClassifier
• Manages FIFO history of fingertip positions
• Loads point_history_classifier.tflite (or .hdf5)
• LSTM/MLP inference → gesture label
d. Post-Processing & Display
• OpenCV GUI window
e. Data-Collection Mode (in app.py)
• On keypress (k or h) captures landmarks/history → appends rows to CSV under model/*/*.csv
f. Offline Training Pipeline
• Jupyter notebooks read CSV → TensorFlow model training → export HDF5 → convert to TFLite → update model directories
Relationships & Data Flow
Webcam → OpenCV Capture → MediaPipe Hands → Landmark Stream →
• KeypointClassifier → hand‐sign result
• PointHistoryClassifier → gesture result
Both results → Overlay & Display → User
• Optional branch:
User command (k/h) → Data‐collector → CSV files
• Separate offline path:
CSV files → Jupyter Notebook → Trainer → HDF5 → TFLite → model directory
Architectural Patterns & Technologies
• Modular pipeline: capture → pose estimation → classification → display
• Edge optimized inference via TFLite
• Separation of concerns:
• app.py (orchestration)
• utils (FPS calc)
• model wrappers (inference)
• notebooks (training)
• Containerization with Docker