How devansh-47/sign-language-to-text-and-speech-conversion works
Project Type & Purpose
• The project is a computer vision based full-stack application (though without a dedicated GUI, it primarily runs Python scripts) that processes real‐time images to detect American sign language hand gestures, classifies them using a convolutional neural network (CNN), and then converts the recognized gesture (or word) into both text and speech.
• Its purpose is to bridge communication for deaf and dumb people by providing a mechanism that translates sign language gestures into written text and audible speech.
Key Directories and Components from the File Structure
• The “AtoZ_3.1” directory contains subdirectories named A through Z. These likely represent datasets or image collections for each alphabet sign used to train or test the CNN.
• Python scripts such as “data_collection_binary.py”, “data_collection_final.py”, “final_pred.py”, and “prediction_wo_gui.py” indicate separate modules for data acquisition, preprocessing or collection, and prediction/inference logic.
• The “cnn8grps_rad1_model.h5” file is the trained Keras CNN model.
Main System Components and Their Relationships
a. Data Acquisition Module:
• Handles input from the webcam.
• Uses media pipe library for extracting hand landmarks.
• Prepares raw input images.
b. Pre-processing & Feature Extraction Module:
• Uses OpenCV to apply filters, convert images to grayscale, apply blurring and thresholding.
• Uses the output of mediapipe (landmark points) to generate simplified images free from background issues.
• Prepares region-of-interest (ROI) images for the CNN.
c. CNN-based Gesture Classification Module:
• The preprocessed gesture images are fed into a CNN (built with Keras/Tensorflow).
• The CNN first classifies gestures into grouped classes (8 groups) and then further refines the classification into the exact alphabet.
• Stores and loads the trained model from “cnn8grps_rad1_model.h5”.
d. Post-Processing and Conversion Module:
• Once a gesture is classified, the recognized alphabet/word is output as text.
• Uses the “pyttsx3” library (or a similar text-to-speech package) to convert text into speech.
e. User Interaction / Interface:
• Even though there might not be a complete GUI, the system interacts with the user via the command line or minimal user display where prediction outputs are shown.
f. Data Storage/Organization:
• The “AtoZ_3.1” folder serves as storage for training/testing images (organized by sign letter), while Python scripts manage the flow from data capture to prediction.
Architectural Patterns and Design Principles
• A modular pipeline architecture where each module is responsible for a specific task (data capture, preprocessing, classification, and post-processing).
• The use of directional data flow is important – data moves sequentially from acquisition to final speech output.
• Separation of concerns: The system’s image preprocessing, CNN model inference, and text-to-speech conversion are clearly isolated.
• High reusability of helper modules such as the mediapipe-based hand detection and OpenCV pre-processing functions.
Adding Specific Notations
– The lower layers include physical devices and raw datasets.
– The mid layers include image processing and machine learning.
– The top layers handle user interaction and output (text and speech).