How dev-d-gr8/signsense works
Project Type and Purpose
• This is a machine-learning pipeline for real-time ASL recognition and sentence construction. It has two main modes:
• Training mode – building a transformer model from prerecorded data
• Inference mode – live video capture → gesture recognition → sentence generation
File Structure
• Top-level files:
• 10042041.parquet – tabular dataset for training
• ASL_Model_Training.ipynb – Jupyter notebook orchestrating the training pipeline
• app.py – main script for live inference (video capture and translation)
• requirements.txt – Python dependencies
• README.md, LICENSE.md – docs
• No separate “src/” or “frontend/” module—both training and serving live in single repo.
Extract Architecture Insights from README
• Core libraries:
• OpenCV & MediaPipe: real-time video capture and hand landmark detection
• TensorFlow: building/testing the transformer model
• Gemini-Pro LLM API: constructs full sentences from recognized ASL tokens
• Real-time pipeline: video → preprocessing → inference → LLM → text output
Main Components & Their Interactions
a. Training Subsystem
• Data Loader (reads 10042041.parquet)
• Preprocessing (data augmentation / MediaPipe offline)
• Model Trainer (Transformer architecture in TensorFlow)
• Model Artifacts (saved weights)
b. Inference Subsystem
• Video Capture Module (OpenCV)
• Preprocessing Module (MediaPipe hand/pose extraction)
• Inference Engine (loads trained Transformer weights + TensorFlow)
• Sentence Constructor (Gemini-Pro LLM API)
• Output Renderer (console/window overlay)
c. External Dependencies
• Gemini-Pro LLM API (external service)
• Python environment (managed via requirements.txt)
Relationships & Data Flows
• Training Flow: Dataset → Notebook → Model Artifacts
• Inference Flow: Camera → Preprocessor → Model Inference → LLM API → UI
Architectural Patterns & Principles
• Pipeline Pattern: clearly separates capture, preprocess, inference, postprocess
• Modular Components: each major step (capture, preprocessing, inference, API call) is a distinct block
• Single Process Monolith: no microservices; all live functionality runs in app.py
• Offline vs. Online Modes: training “batch” vs. inference “streaming”