# gabguerin/sign-language-recognition--mediapipe-dtw architecture

> Architecture diagram of the GitHub repository gabguerin/sign-language-recognition--mediapipe-dtw, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gabguerin/sign-language-recognition--mediapipe-dtw
- Repository: https://github.com/gabguerin/sign-language-recognition--mediapipe-dtw

## Overview

##### Project type and purpose
- This is a stand-alone computer-vision/ML pipeline in Python for real-time Sign Language Recognition.
- It uses MediaPipe for landmark extraction and Dynamic Time Warping (DTW) for time-series similarity.

##### File structure
- data/
– videos/ (raw sign videos, organized by label)
– dataset/ (preprocessed or placeholder for processed data)
- main.py (orchestrator for loading model, starting webcam, handling user input)
- sign_recorder.py (captures, processes, and predicts signs via DTW + voting)
- yt_download.py + yt_links.csv (automated dataset creation via YouTube + ffmpeg)
- models/
– hand_model.py (encapsulates MediaPipe hand landmarks → feature vectors)
– pose_model.py, sign_model.py (structure for pose or sign sequences)
- utils/ (helper functions)
– mediapipe_utils.py (init & run MediaPipe Holistic)
– landmark_utils.py (normalize/rotate/scale landmarks)
– dataset_utils.py (file I/O, preprocessing pipelines)
– dtw.py (DTW distance computation)
- webcam_manager.py (abstraction over OpenCV webcam feed)
- requirements.txt (dependencies: MediaPipe, OpenCV, NumPy, fastdtw, etc.)

##### README for additional insights
- Workflow: ingest videos → extract landmarks → compute angles (invariant to scale/orientation) → build per-sign models → at runtime record gesture → compute DTW against all reference signs → voting & thresholding → output label.

##### Elements
- Clear labels for each block (e.g., “MediaPipe Utils,” “HandModel Feature Extraction,” “DTW Engine”).
- Blue for offline/batch components
- Green for real-time/inference components
- Orange for external services
- Gray for data stores (video folders, model files)
- Rectangles for code modules
- Cloud/music-note for YouTube/ffmpeg external service

##### Final advice
- Be very detailed: break “Preprocessing” into sub-blocks for dataset_utils, mediapipe_utils, landmark_utils.
- Separate the project into as many independent components as possible to highlight modularity.
- Bidirectional control (e.g., main.py calls webcam_manager & sign_recorder, sign_recorder returns prediction).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Offline Pipeline
    subgraph "Offline Pipeline"
        direction TB
        DS_Local["Local Dataset<br>data/videos/"]:::datastore
        YT_Downloader["YouTube Downloader<br>yt_download.py"]:::offline
        YT_Service(["YouTube & ffmpeg"]):::external

        subgraph "Preprocessing & Feature Extraction"
            direction TB
            DatasetIO["Dataset Utils<br>utils/dataset_utils.py"]:::offline
            MP_Utils["MediaPipe Utils<br>utils/mediapipe_utils.py"]:::offline
            LU_Utils["Landmark Utils<br>utils/landmark_utils.py"]:::offline
        end

        subgraph "Model Representation & Storage"
            direction TB
            HandModel["HandModel Definition<br>models/hand_model.py"]:::offline
            PoseModel["PoseModel Structure<br>models/pose_model.py"]:::offline
            SignModel["SignModel Storage<br>models/sign_model.py & data/dataset/"]:::datastore
        end
    end

    %% Real-Time Recognition
    subgraph "Real-Time Recognition"
        direction TB
        WebcamMgr["Webcam Manager<br>webcam_manager.py"]:::realtime
        Main["Main Orchestrator<br>main.py"]:::realtime
        SignRec["Sign Recorder<br>sign_recorder.py"]:::realtime
        DTW["DTW Engine<br>utils/dtw.py"]:::realtime
        UI["Output Overlay"]:::realtime
    end

    %% Offline flow
    YT_Service -->|"downloads videos"| YT_Downloader
    YT_Downloader -->|"stores in"| DS_Local
    DS_Local -->|"reads videos"| DatasetIO
    DatasetIO -->|"extracts landmarks"| MP_Utils
    MP_Utils -->|"normalizes landmarks"| LU_Utils
    LU_Utils -->|"produces feature vectors"| HandModel
    LU_Utils -->|"produces pose sequences"| PoseModel
    HandModel -->|"builds reference"| SignModel
    PoseModel -->|"builds reference"| SignModel

    %% Real-time flow
    WebcamMgr -->|"frame stream"| Main
    Main -->|"capture request"| SignRec
    SignRec -->|"feature vectors"| DTW
    DTW -->|"compares with references"| SignModel
    SignRec -->|"prediction"| Main
    Main -->|"displays result"| UI

    %% Click Events
    click DS_Local "https://github.com/gabguerin/sign-language-recognition--mediapipe-dtw/tree/master/data/videos/"
    click YT_Downloader "https://github.com/gabguerin/sign-language-recognition--mediapipe-dtw/blob/master/yt_download.py"
    click DatasetIO "https://github.com/gabguerin/sign-language-recognition--mediapipe-dtw/blob/master/utils/dataset_utils.py"
    click MP_Utils "https://github.com/gabguerin/sign-language-recognition--mediapipe-dtw/blob/master/utils/mediapipe_utils.py"
    click LU_Utils "https://github.com/gabguerin/sign-language-recognition--mediapipe-dtw/blob/master/utils/landmark_utils.py"
    click HandModel "https://github.com/gabguerin/sign-language-recognition--mediapipe-dtw/blob/master/models/hand_model.py"
    click PoseModel "https://github.com/gabguerin/sign-language-recognition--mediapipe-dtw/blob/master/models/pose_model.py"
    click SignModel "https://github.com/gabguerin/sign-language-recognition--mediapipe-dtw/blob/master/models/sign_model.py"
    click WebcamMgr "https://github.com/gabguerin/sign-language-recognition--mediapipe-dtw/blob/master/webcam_manager.py"
    click Main "https://github.com/gabguerin/sign-language-recognition--mediapipe-dtw/blob/master/main.py"
    click SignRec "https://github.com/gabguerin/sign-language-recognition--mediapipe-dtw/blob/master/sign_recorder.py"
    click DTW "https://github.com/gabguerin/sign-language-recognition--mediapipe-dtw/blob/master/utils/dtw.py"

    %% Styles
    classDef offline fill:#bbdefb,stroke:#1e88e5,color:#0d47a1
    classDef realtime fill:#c8e6c9,stroke:#43a047,color:#1b5e20
    classDef external fill:#ffcc80,stroke:#fb8c00,color:#e65100
    classDef datastore fill:#cfd8dc,stroke:#607d8b,color:#263238
```
