How sooryak12/indian-sign-language-recognition works
Project type and purpose
• This is a full-stack machine-learning-powered web/CLI application for word-level Indian Sign Language recognition from video.
• Purpose: ingest a video (live camera feed, uploaded video, or file), extract pose keypoints via Mediapipe PoseNet, feed these into an LSTM model, and return a predicted word.
File structure
• Top-level:
• app.py – Flask (or similar) web server exposing HTTP API & web UI
• deploy_code.py – live-camera inference script
• run-through-cmd-line.py – CLI video-file inference script
• helper_functions.py – shared utilities (frame extraction, keypoint processing)
• requirements.txt
• lstm-model/ – trained LSTM model weights (.hdf5)
• crnn-model-v1-initial-attempt-files/ – legacy CRNN experiments (not part of final deployment)
• training-data/ – raw and augmented videos + data.txt
• The README describes two architectures (CRNN v1 and final LSTM v2) and the data pipeline.
Extracted insights from README
• Core inference pipeline:
Video feed → select 45 frames → Mediapipe PoseNet → numpy keypoint array (45×24×2) → LSTM model → predicted word.
• Web app exposes endpoints: POST /upload-video/ and GET /test/, driven by app.py.
• Two deployment modes: web server (app.py) and CLI/live camera (deploy_code.py, run-through-cmd-line.py).
Main components
a. Clients:
• Web browser/mobile front end
• CLI tool or direct Python script (live camera / file)
b. API Layer (Flask server – app.py):
• /upload-video endpoint
• /test endpoint
c. Video Ingestion Module:
• Live camera capture (deploy_code.py)
• File upload handler
• CLI file reader (run-through-cmd-line.py)
d. Preprocessing Module:
• Frame selector (45 frames logic)
• Mediapipe PoseNet (external library/service)
• Keypoint extraction & packaging into numpy array
e. Inference Module:
• LSTM model loader (lstm-model/170-0.83.hdf5)
• Prediction engine
f. Results / Response:
• JSON/HTML response with predicted word
• Optionally log or save results
g. Model Training Pipeline (optional, for completeness):
• training-data/
• Notebooks (train-model.ipynb, crnn-model notebooks)
• produce new weights under lstm-model/ or crnn-model-v1
Relationships and interactions
• Clients → (HTTP / CLI invocation) → API Layer or direct script
• API Layer → Video Ingestion → Preprocessing → Inference → API Layer → Client
• Preprocessing calls Mediapipe PoseNet (depicted as an external service or library block).
• Inference uses model artifacts stored in “lstm-model” (storage block).
• Training Pipeline is separate: training-data → notebooks/scripts → model weights → model storage.
Architectural patterns & design principles
• Separation of concerns:
• Interface layer (Flask / CLI)
• Preprocessing layer
• Inference layer
• Model & data storage
• Stateless inference service: each request is independent, model loaded in memory.
• Modular code organization: helper_functions.py, separate scripts for deployment modes.
• Reproducibility: explicit requirements.txt, versioned model directories.