How tanmayjivnani/sign-language-to-text-and-speech works
Main Components
• Dataset Collection Module: Represented by the script "collectImgs.py" which is responsible for capturing gesture images from the webcam.
• Data Preprocessing and Feature Extraction Module: This is handled by "createDataset.py" which extracts 42 features per sample (using MediaPipe for hand landmarks) and saves them in a pickle file.
• Model Training Module: Represented by "trainClassifier.py" which takes the preprocessed dataset and trains a Random Forest Classifier. Also note that “model.p” is the pre-trained model file in the repository.
• Inference / Real-Time Gesture Recognition Module: The "main.py" script integrates the previously trained model, processes real-time input from a webcam, stabilizes predictions using a stabilization buffer, and then converts the recognized gestures into words/sentences.
• Text-to-Speech (TTS) and User Interface: Alongside the recognition, a TTS engine (using Pyttsx3) and a Tkinter based GUI display the current alphabet, word, and sentence for user feedback.
Relationships and Component Interactions
• Input Flow: Start with the webcam. The image feed is captured in the Dataset Collection stage.
• Preprocessing: Images are then processed using MediaPipe (extracting key landmarks) in the Data Preprocessing component.
• Training: The processed features are fed into a Machine Learning Trainer (using the Random Forest Classifier) which produces a trained model.
• Inference: In real-time, the webcam feed is preprocessed in a similar manner, then the ML model (pre-trained or custom-trained) recognizes the gestures.
• Stabilization & Interpretation: A stabilization buffer smooths out recognition errors and then the recognized gesture is converted into readable text.
• Output: The recognized text is both displayed on the GUI and converted into speech using TTS.
Important Architectural Patterns/Design Principles
• Modular Decomposition: Separate scripts for dataset collection, preprocessing, model training, and inference imply a modular design that can be individually maintained or upgraded.
• Data Flow Pipeline: There is clearly a pipeline starting from data acquisition, feature extraction, model building, and finally real-time inference.
• Reusability and Extensibility: The inclusion of training scripts, a pre-trained model, and the potential for future enhancements (dynamic gestures, mobile deployment) underscore the design’s adaptability.
• Real-Time Processing: Emphasis on stabilization and real-time processing is a crucial architectural component to ensure accuracy during inference.
Relevant Technologies and Frameworks
• Languages & Libraries: Python, MediaPipe, OpenCV (for image capture and processing), Scikit-learn (for training the Random Forest model).
• GUI: Tkinter is used for building a graphical interface.
• TTS: Pyttsx3 is used for converting text to speech.
• Data Persistence: Use of pickle files for storing processed features and the pre-trained model.
Specific Elements
• From Webcam → Dataset Collection → Preprocessing (MediaPipe Feature Extraction)
• From Preprocessed Data → Model Training → Producing A Model (model.p)
• From Webcam (inference loop) → Preprocessing → ML Prediction → Stabilization Buffer
• From Stabilization → Text-Conversion → GUI & TTS
• Adding annotations that reference the supporting libraries or technologies (e.g., a note that "MediaPipe extracts 21 hand landmarks for 42 features").