# sooryak12/indian-sign-language-recognition architecture

> Architecture diagram of the GitHub repository sooryak12/indian-sign-language-recognition, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sooryak12/indian-sign-language-recognition
- Repository: https://github.com/sooryak12/indian-sign-language-recognition

## Overview

##### Project type and purpose
- This is a full-stack machine-learning-powered web/CLI application for word-level Indian Sign Language recognition from video.
- Purpose: ingest a video (live camera feed, uploaded video, or file), extract pose keypoints via Mediapipe PoseNet, feed these into an LSTM model, and return a predicted word.

##### File structure
- Top-level:
- app.py           – Flask (or similar) web server exposing HTTP API & web UI
- deploy_code.py   – live-camera inference script
- run-through-cmd-line.py – CLI video-file inference script
- helper_functions.py – shared utilities (frame extraction, keypoint processing)
- requirements.txt
- lstm-model/      – trained LSTM model weights (.hdf5)
- crnn-model-v1-initial-attempt-files/ – legacy CRNN experiments (not part of final deployment)
- training-data/   – raw and augmented videos + data.txt
- The README describes two architectures (CRNN v1 and final LSTM v2) and the data pipeline.

##### Extracted insights from README
- Core inference pipeline:
Video feed → select 45 frames → Mediapipe PoseNet → numpy keypoint array (45×24×2) → LSTM model → predicted word.
- Web app exposes endpoints: POST /upload-video/ and GET /test/, driven by app.py.
- Two deployment modes: web server (app.py) and CLI/live camera (deploy_code.py, run-through-cmd-line.py).

##### Main components
a. Clients:
- Web browser/mobile front end
- CLI tool or direct Python script (live camera / file)
b. API Layer (Flask server – app.py):
- /upload-video endpoint
- /test endpoint
c. Video Ingestion Module:
- Live camera capture (deploy_code.py)
- File upload handler
- CLI file reader (run-through-cmd-line.py)
d. Preprocessing Module:
- Frame selector (45 frames logic)
- Mediapipe PoseNet (external library/service)
- Keypoint extraction & packaging into numpy array
e. Inference Module:
- LSTM model loader (lstm-model/170-0.83.hdf5)
- Prediction engine
f. Results / Response:
- JSON/HTML response with predicted word
- Optionally log or save results
g. Model Training Pipeline (optional, for completeness):
- training-data/
- Notebooks (train-model.ipynb, crnn-model notebooks)
- produce new weights under lstm-model/ or crnn-model-v1

##### Relationships and interactions
- Clients → (HTTP / CLI invocation) → API Layer or direct script
- API Layer → Video Ingestion → Preprocessing → Inference → API Layer → Client
- Preprocessing calls Mediapipe PoseNet (depicted as an external service or library block).
- Inference uses model artifacts stored in “lstm-model” (storage block).
- Training Pipeline is separate: training-data → notebooks/scripts → model weights → model storage.

##### Architectural patterns & design principles
- Separation of concerns:
- Interface layer (Flask / CLI)
- Preprocessing layer
- Inference layer
- Model & data storage
- Stateless inference service: each request is independent, model loaded in memory.
- Modular code organization: helper_functions.py, separate scripts for deployment modes.
- Reproducibility: explicit requirements.txt, versioned model directories.

## Diagram (Mermaid)

```mermaid
graph TB
    %% User-facing components
    WebBrowser["Web Browser/Mobile Front End"]:::user
    CLI["CLI Tool / Python Script"]:::user

    %% API Layer
    API["API Layer<br>(Flask Server)"]:::user
    click API "https://github.com/sooryak12/indian-sign-language-recognition/blob/pycode/app.py"

    %% Video Ingestion Module
    subgraph "Video Ingestion Module"
        direction TB
        LiveCamera["Live Camera Capture"]:::ml
        click LiveCamera "https://github.com/sooryak12/indian-sign-language-recognition/blob/pycode/deploy-code.py"
        FileUploader["File Upload Handler"]:::ml
        CLIReader["CLI File Reader"]:::ml
        click CLIReader "https://github.com/sooryak12/indian-sign-language-recognition/blob/pycode/run-through-cmd-line.py"
    end

    %% Preprocessing Module
    subgraph "Preprocessing Module"
        direction TB
        FrameSelector["Frame Selector<br>(45 frames)"]:::ml
        PoseNet["Mediapipe PoseNet"]:::ext
        KeypointExtractor["Keypoint Extraction & Packaging"]:::ml
        click KeypointExtractor "https://github.com/sooryak12/indian-sign-language-recognition/blob/pycode/helper_functions.py"
    end

    %% Inference Module
    subgraph "Inference Module"
        direction TB
        ModelLoader["LSTM Model Loader"]:::ml
        click ModelLoader "https://github.com/sooryak12/indian-sign-language-recognition/blob/pycode/lstm-model/170-0.83.hdf5"
        Predictor["Prediction Engine"]:::ml
    end

    %% Storage Components
    ModelStorage["LSTM Model Storage (Weights)"]:::storage
    click ModelStorage "https://github.com/sooryak12/indian-sign-language-recognition/blob/pycode/lstm-model/170-0.83.hdf5"
    TrainingData["Training Data Storage"]:::storage
    click TrainingData "https://github.com/sooryak12/indian-sign-language-recognition/tree/pycode/training-data/"

    %% Training Pipeline (Offline)
    subgraph "Training Pipeline (Offline)"
        direction TB
        TrainingNotebook["Train Model Notebook"]:::ml
        click TrainingNotebook "https://github.com/sooryak12/indian-sign-language-recognition/blob/pycode/train-model.ipynb"
        LegacyCRNN["Legacy CRNN Experiments"]:::ml
        click LegacyCRNN "https://github.com/sooryak12/indian-sign-language-recognition/tree/pycode/crnn-model-v1-initial-attempt-files/"
    end

    %% Data Flow
    WebBrowser -->|"HTTP MP4 video"| API
    CLI -->|"Video input"| CLIReader
    API -->|"via upload"| FileUploader
    LiveCamera -->|"camera feed"| FrameSelector
    FileUploader -->|"uploaded video"| FrameSelector
    CLIReader -->|"video file"| FrameSelector

    FrameSelector -->|"raw frames"| PoseNet
    PoseNet -->|"keypoints"| KeypointExtractor
    KeypointExtractor -->|"45×24×2 numpy array"| ModelLoader
    ModelLoader -->|"loads weights"| ModelStorage
    ModelLoader --> Predictor
    Predictor -->|"predicted word"| API
    API -->|"HTTP JSON/HTML response"| WebBrowser

    %% Training Flow
    TrainingData -->|"videos + labels"| TrainingNotebook
    TrainingNotebook -->|"new weights"| ModelStorage
    TrainingData --> LegacyCRNN

    %% Styles
    classDef user fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef ml fill:#90EE90,stroke:#333,stroke-width:1px
    classDef ext fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef storage fill:#E0FFE0,stroke:#333,stroke-width:1px
```
