How shag527/indian-sign-language-recognition works
Project Type and Purpose
• This project is a full-stack recognition system that processes Indian Sign Language gestures. It involves collecting hand gesture images, preprocessing them using both background subtraction and skin segmentation, extracting SURF features using a Bag-of-Words approach, classifying the gestures with several machine learning models (KNN, SVM, logistic regression, Naive Bayes, CNN), and providing outputs as both text and speech.
• Dual modes are supported: one for regular sign-based input and one for reverse sign recognition (processing spoken words and displaying corresponding sign images).
File Structure
• The top-level “Code” folder is divided into several functional modules:
• Bag of Features – handles histogram generation from SURF descriptors.
• Classification – contains various classifier implementations (CNN, KNN, SVM, logistic regression, naive bayes) along with their dataset CSV files.
• Predict signs – integrates the prediction functionality, reverse recognition logic, dataset creation, and a main tkinter UI.
• Preprocessing – contains notebooks that perform dataset division and image preprocessing.
• Additional folders such as “Images” (for sample datasets and output visuals) and “Scripts” (environment activation scripts) support deployment and running the project.
Extracting Details from the README
• The README outlines the data workflow:
• Getting started instructions including dependency management.
• Preprocessing using background subtraction and skin segmentation.
• Feature extraction using SURF and clustering via MiniKbatchMeans to create a bag of visual words histogram.
• Employing multiple classifiers followed by accuracy analysis.
• A dual-mode approach to support both text-to-speech conversion and reverse sign recognition using speech-to-text via Google Speech API.
• It also mentions the user interaction via a tkinter GUI, where users log in, choose services, or create their own sign dataset.
Core Components
a. Modules/Components:
• Data Input & Dataset: Representing the manually created dataset (images) accessible via the “Images” folder.
• Preprocessing Module: Split into subcomponents running in Jupyter notebooks (background subtraction and skin segmentation). This module processes raw images to filter out background/noise.
• Feature Extraction Module (Bag of Features): Uses SURF to extract and cluster features and then generate histograms as CSV files.
• Classification Module: Contains various ML classifiers (CNN, KNN, SVM, Logistic Regression, Naive Bayes) with CSV data input and output predictions.
• Prediction & Reverse Recognition Module: Incorporates the prediction logic, reverse recognition using speech input (Google Speech API) and displaying results through a tkinter-based UI.
• Output & Communication Module: Handles converting the predictions to text and speech output (via Pyttsx3) and updating the GUI.
b. Relationships & Interactions:
• A clear linear data flow exists starting from raw image acquisition → Preprocessing → Feature Extraction → Classification, with the outputs funneled into the UI for both sign prediction and reverse sign recognition.
• There is also a feedback loop for creating/updating the dataset and retraining the classifiers.
• The UI component (main.py in “Predict signs”) interacts with the classification modules and then triggers appropriate output functionality.
c. Architectural Patterns & Design Principles:
• Modular design: Each functional aspect (preprocessing, feature extraction, classification) is divided into separate directories to ensure loose coupling.
• Layered Architecture: Separation of preprocessing/feature extraction from classification and UI.
• Integration of machine learning pipelines with user interface components.
d. Key Technologies & Frameworks:
• Programming Language: Python (3.7.1)
• Libraries: OpenCV (and opencv-contrib for SURF), scikit-learn (for classifiers like KNN, SVM, etc.), TensorFlow/Keras or similar for CNN, Pyttsx3 for text to speech, and tkinter for GUI.
• Tools for environment/ dependency management: pip, requirements.txt, and conda.
Specific Elements
• Data flow from the raw input to preprocessing → feature extraction → classification → UI output.
• Interactions between the GUI and backend processing.
• For data processing components (use rectangles),
• Incorporate annotations where necessary, such as indicating that the “Bag of Features” module outputs CSV files used by classifier scripts.