# kazuhito00/hand-gesture-recognition-using-mediapipe architecture

> Architecture diagram of the GitHub repository kazuhito00/hand-gesture-recognition-using-mediapipe, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kazuhito00/hand-gesture-recognition-using-mediapipe
- Repository: https://github.com/kazuhito00/hand-gesture-recognition-using-mediapipe

## Overview

##### Project Type and Purpose
- This is a standalone computer-vision/ML application (not a web or full-stack service).
- It uses MediaPipe (Python) to extract hand landmarks from a webcam feed, then classifies hand poses and finger‐motion gestures using two separate lightweight ML models (TFLite/HDF5).
- It also includes data‐collection modes and Jupyter notebooks for retraining those models.

##### File Structure
- Top-level:
- app.py                → main demo & data‐collection driver
- keypoint_classification.ipynb, point_history_classification.ipynb
- requirements.txt, Dockerfile, .dockerignore
- model/
├─ keypoint_classifier/     → CSV dataset, inference wrapper, HDF5 & TFLite hand‐sign model, labels
└─ point_history_classifier/→ CSV dataset, inference wrapper, HDF5 & TFLite finger‐motion model, labels
- utils/
└─ cvfpscalc.py             → FPS calculation helper

##### Core Components
a. Input Layer
- Webcam Device (OpenCV)

b. Preprocessing & Landmark Extraction
- MediaPipe Hands pipeline → outputs a stream of 21 keypoints

c. Inference Modules
1. KeypointClassifier
- Loads keypoint_classifier.tflite (or .hdf5)
- Preprocesses landmarks → MLP inference → hand‐sign label
2. PointHistoryClassifier
- Manages FIFO history of fingertip positions
- Loads point_history_classifier.tflite (or .hdf5)
- LSTM/MLP inference → gesture label

d. Post-Processing & Display
- OpenCV GUI window

e. Data-Collection Mode (in app.py)
- On keypress (k or h) captures landmarks/history → appends rows to CSV under model/*/*.csv

f. Offline Training Pipeline
- Jupyter notebooks read CSV → TensorFlow model training → export HDF5 → convert to TFLite → update model directories

##### Relationships & Data Flow
Webcam → OpenCV Capture → MediaPipe Hands → Landmark Stream →
- KeypointClassifier → hand‐sign result
- PointHistoryClassifier → gesture result
Both results → Overlay & Display → User

- Optional branch:
User command (k/h) → Data‐collector → CSV files

- Separate offline path:
CSV files → Jupyter Notebook → Trainer → HDF5 → TFLite → model directory

##### Architectural Patterns & Technologies
- Modular pipeline: capture → pose estimation → classification → display
- Edge optimized inference via TFLite
- Separation of concerns:
- app.py (orchestration)
- utils (FPS calc)
- model wrappers (inference)
- notebooks (training)
- Containerization with Docker

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Docker Container"
        direction TB

        InputLayer["Webcam Device\n(OpenCV)"]:::input
        click InputLayer "https://github.com/kazuhito00/hand-gesture-recognition-using-mediapipe/blob/main/app.py"

        Preprocessing["MediaPipe Hands\n(Landmark Extraction)"]:::preprocess
        click Preprocessing "https://github.com/kazuhito00/hand-gesture-recognition-using-mediapipe/blob/main/app.py"

        InputLayer -->|"Video Frames"| Preprocessing

        subgraph "Inference Modules"
            direction TB
            KPWrapper["KeypointClassifier\n(wrapper)"]:::inference
            click KPWrapper "https://github.com/kazuhito00/hand-gesture-recognition-using-mediapipe/blob/main/model/keypoint_classifier/keypoint_classifier.py"
            PHWrapper["PointHistoryClassifier\n(wrapper)"]:::inference
            click PHWrapper "https://github.com/kazuhito00/hand-gesture-recognition-using-mediapipe/blob/main/model/point_history_classifier/point_history_classifier.py"
        end

        Preprocessing -->|"Landmark List"| KPWrapper
        Preprocessing -->|"Landmark List"| PHWrapper

        %% KeypointClassifier model files
        KP_TFLite["keypoint_classifier.tflite"]:::storage
        click KP_TFLite "https://github.com/kazuhito00/hand-gesture-recognition-using-mediapipe/blob/main/model/keypoint_classifier/keypoint_classifier.tflite"
        KP_HDF5["keypoint_classifier.hdf5"]:::storage
        click KP_HDF5 "https://github.com/kazuhito00/hand-gesture-recognition-using-mediapipe/blob/main/model/keypoint_classifier/keypoint_classifier.hdf5"
        KP_Label["keypoint_classifier_label.csv"]:::storage
        click KP_Label "https://github.com/kazuhito00/hand-gesture-recognition-using-mediapipe/blob/main/model/keypoint_classifier/keypoint_classifier_label.csv"
        KPWrapper -->|"Loads Model & Labels"| KP_TFLite
        KPWrapper --> KP_HDF5
        KPWrapper --> KP_Label
        %% data-collection CSV
        KP_CSV["keypoint.csv"]:::storage
        click KP_CSV "https://github.com/kazuhito00/hand-gesture-recognition-using-mediapipe/blob/main/model/keypoint_classifier/keypoint.csv"

        PH_TFLite["point_history_classifier.tflite"]:::storage
        click PH_TFLite "https://github.com/kazuhito00/hand-gesture-recognition-using-mediapipe/blob/main/model/point_history_classifier/point_history_classifier.tflite"
        PH_HDF5["point_history_classifier.hdf5"]:::storage
        click PH_HDF5 "https://github.com/kazuhito00/hand-gesture-recognition-using-mediapipe/blob/main/model/point_history_classifier/point_history_classifier.hdf5"
        PH_Label["point_history_classifier_label.csv"]:::storage
        click PH_Label "https://github.com/kazuhito00/hand-gesture-recognition-using-mediapipe/blob/main/model/point_history_classifier/point_history_classifier_label.csv"
        PHWrapper --> PH_TFLite
        PHWrapper --> PH_HDF5
        PHWrapper --> PH_Label
        PH_CSV["point_history.csv"]:::storage
        click PH_CSV "https://github.com/kazuhito00/hand-gesture-recognition-using-mediapipe/blob/main/model/point_history_classifier/point_history.csv"

        %% Branch inference outputs
        KPWrapper -->|"Hand-Sign Label"| Overlay
        PHWrapper -->|"Gesture Label"| Overlay

        Overlay["Overlay & Display\n(OpenCV GUI)"]:::output
        click Overlay "https://github.com/kazuhito00/hand-gesture-recognition-using-mediapipe/blob/main/app.py"
        FPSHelper["FPS Calculator"]:::utility
        click FPSHelper "https://github.com/kazuhito00/hand-gesture-recognition-using-mediapipe/blob/main/utils/cvfpscalc.py"
        Overlay -->|"Renders Frame with\nLabels & FPS"| User

        User["User\n(Console/Keypress)"]:::external

        %% Data collection branch
        User -->|"keypress k/h"| DataCollector
        DataCollector["Data Collection\n(app.py)"]:::process
        click DataCollector "https://github.com/kazuhito00/hand-gesture-recognition-using-mediapipe/blob/main/app.py"
        DataCollector -->|"Append Rows"| KP_CSV
        DataCollector -->|"Append Rows"| PH_CSV
    end

    subgraph "Offline Training Pipeline" 
        direction TB
        CSVFiles["CSV Datasets"]:::storage
        CSVFiles --> Notebooks
        Notebooks["Jupyter Notebooks"]:::offline
        click Notebooks "https://github.com/kazuhito00/hand-gesture-recognition-using-mediapipe/blob/main/keypoint_classification.ipynb"
        click Notebooks "https://github.com/kazuhito00/hand-gesture-recognition-using-mediapipe/blob/main/point_history_classification.ipynb"
        Notebooks -->|"Train TensorFlow\n(Export HDF5)"| OfflineHDF5
        OfflineHDF5["Model.hdf5"]:::storage
        Notebooks -->|"Convert to TFLite"| OfflineTFLite
        OfflineTFLite["Model.tflite"]:::storage
        OfflineTFLite -->|"Update Model Dir"| ModelDirs
        ModelDirs["model/*"]:::storage
    end

    subgraph "Containerization" 
        direction TB
        Dockerfile["Dockerfile"]:::infra
        Dockerignore[".dockerignore"]:::infra
    end

    classDef input fill:#AEDFF7,stroke:#0366D6,color:#0366D6;
    classDef preprocess fill:#FFD8A8,stroke:#FF8C00,color:#8B4000;
    classDef inference fill:#C6F6D5,stroke:#2F855A,color:#22543D;
    classDef output fill:#E9D8FD,stroke:#805AD5,color:#553C9A;
    classDef utility fill:#EDF2F7,stroke:#4A5568,color:#2D3748;
    classDef process fill:#FFF5F7,stroke:#B83280,color:#97266D;
    classDef storage fill:#F0FFF4,stroke:#38A169,color:#2F855A;
    classDef offline fill:#F7FAFC,stroke:#718096,color:#2D3748;
    classDef infra fill:#F7FAFC,stroke:#A0AEC0,color:#2D3748;
```
