How tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation works
Project Type and Purpose
• This project is a deep-learning based real-time recognition system. It uses image capture (likely from a webcam) to detect Indian Sign Language gestures and translates the recognized alphabets into Braille script.
• The project is structured as a research/prototype or experimental ML pipeline built using Jupyter notebooks, model files (Keras models, TensorFlow saved models, and TFLite models), and multiple directories for outputs and results.
File Structure to Discover Major Components
• Data Acquisition:
– Notebooks like CapturePicture.ipynb and CollectData.ipynb indicate that images are captured from a live feed or camera.
– The "New pictures" and "Pictures" directories contain image data.
• Data Preparation:
– DatasetPrepare.ipynb is used to prepare the dataset for training or validation.
• Model Training and Benchmarking:
– Multiple notebooks (PrepareModelBenchmark.ipynb, PrepareModelCustom.ipynb) suggest differing approaches to model training (benchmark vs. custom).
– The “models” directory contains pre-trained or reference model files (e.g., actionCO.h5, actionMN.h5, etc.) and weights.
– The presence of numerous saved model directories (saved_models, saved_models_benchmark, etc.) shows that models are being persisted in different formats.
• Inference and Application:
– The app.ipynb notebook is likely used for running real-time inferences and displaying the Braille translation outputs.
• Output and Results:
– Directories like "Output" and "Results" (including accuracy, loss, and architecture PDFs) indicate analysis and visualization phases.
– There is an existing architecture_diagram (both .drawio and .pdf) that could serve as a reference or be updated.
• Export Models for Edge Deployment:
– The tflite_models directory contains TensorFlow Lite models, suggesting a focus on deploying lightweight inference on mobile or embedded devices.
a. Main Components:
– Input Module: Camera/Data Capture Environment
– Data Collection & Storage: Handling new images, dataset directories, and image preprocessing.
– Data Preparation & Preprocessing Module: Using DatasetPrepare.ipynb to normalize and format images.
– Model Training & Evaluation Module:
• Two paths: Benchmark model and Custom model (each with its own training logic and saved weights).
– Model Deployment Module:
• Inference engine in app.ipynb that loads trained models from either “models”, “saved_models”, or the optimized TFLite models.
– Translation Engine:
• A subcomponent that maps the recognized sign language (English alphabet) to its corresponding Braille script.
– Output Display:
• Real-time display through the app, along with saving output PDFs and logs.
b. Relationships and Data Flow:
– The Input Module sends captured images to the Data Preparation Module.
– Preprocessed data is then fed into the Model Training Module for both benchmark and custom models.
– Trained models are saved to “saved_models” directories, and optionally converted to TFLite format.
– In real-time operation, the Inference Engine receives new camera input, processes it using the trained model, and passes the recognized sign to the Translation Engine.
– The system outputs the Braille translation and may update the UI or save logs (in “Output” or “Results” folders).
c. Architectural Patterns and Design Principles:
– Modular design: Each notebook and directory points to a separation of concerns (capture, preprocess, train, infer, output).
– Data pipeline structure: Raw data → preprocessing → model training → inference → translation.
– Experimentation logging: Separate outputs for benchmarks and custom models for performance evaluation.
– Model portability: Use of both Keras/TensorFlow models and TFLite for embedded deployment.
d. Relevant Technologies:
– Python and Jupyter Notebooks for prototyping.
– Deep Learning frameworks (TensorFlow/Keras) suggested by .h5 and saved_model directories.
– TensorFlow Lite for mobile/edge deployment.