How alpacamoon/sign-language-recognition works
Project Type & Purpose
• This is a local, full-featured Python application built with KivyMD that captures American Sign Language (ASL) via a webcam.
• It processes both static (alphabet and numbers) and dynamic (word gestures) signs and produces text and/or audio outputs.
• In addition, the project incorporates post‐processing for sentence generation (to produce fluent outputs), language translation, and text-to-speech conversion.
Analyzing the File Structure
• There are distinct modules/directories that each represent major components of the overall system:
• “kivy_components” and “kivy_main.py”: These support the user interface and camera input using Kivy.
• “action_feature_extraction” and “static_feature_extraction”: These modules contain code for preprocessing video frames and extracting features for dynamic (action-based) signs and static signs respectively.
• “action_recognition” and “static_recognition”: These contain the recognition models (with saved model files and organized keypoints data) that use the extracted features to predict the interpreted sign.
• “sentence_generator”: This module takes recognized individual signs and concatenates them into fluent, human-readable sentences, possibly with additional language models.
• “translation_module”: Handles conversion of the text into other languages using provided JSON language mappings.
• “text_to_speech”: Converts the final text output into audio (e.g., via GTTS or similar libraries).
• “word_segmentation”: Likely helps break or merge recognized characters/words for better contextual accuracy.
• “reference”: Contains supporting files such as sample videos and supplementary CSV/excel reference documents.
• There are also directories for logs, caches, fonts, and requirements which support training, debugging, and dependency management.
Components & Their Relationships
• User Interface (UI) Layer:
• Kivy-based UI (“kivy_main.py” and “kivy_components”) handles video capture and user interactions.
• Input (Webcam Capture):
• Captured frames are sent into two parallel feature extraction pipelines:
▪ Dynamic Processing: “action_feature_extraction” extracts features from video sequences of dynamic signs.
▪ Static Processing: “static_feature_extraction” processes static gestures (alphabets/numbers).
• Recognition Engines:
• “action_recognition”: Uses extracted features to classify dynamic ASL gestures.
• “static_recognition”: Recognizes static sign gestures.
• Both engines depend on pre-trained models and produce recognition outputs (labels, probabilities, etc.).
• Natural Language & Post-Processing Components:
• “sentence_generator”: Consolidates single sign outputs into sentences.
• “word_segmentation”: Refines the segmentation of characters or words to enhance accuracy.
• “translation_module”: Maps recognized text into other languages if required.
• Output Layer:
• “text_to_speech”: Converts final text outputs to audio.
• The system may also display the text result visually within the UI.
• Supporting & Logging Components:
• Log directories (within feature extraction modules) record training/testing events for debugging and model improvement.
• Reference assets (videos, CSVs, Excel files) provide ground truth, training samples, and lookup information.
Key Architectural Patterns & Design Principles
• Modular / Layered Architecture: Each distinct functionality (e.g., recognition, sentence generation, translation) is encapsulated in its own module. This separation promotes maintainability and clearer data flow.
• Data Abstraction: The inputs and outputs between modules are abstracted (for instance, raw video frames to feature vectors, then to recognized sign classes, and finally to sentence and speech output).
• Use of Pre-trained Models: In the recognition modules, specific models are stored (files in “models/”) to perform classification, showing the machine learning component’s integration.
Final Recommendations
"Webcam Input → Feature Extraction → Recognition → Sentence Generation → (Optional Translation) → Text and TTS Output."
• You might also include annotations for any configuration or dependency files (like “requirements.txt”) that hint at external libraries supporting these components.