# alpacamoon/sign-language-recognition architecture

> Architecture diagram of the GitHub repository alpacamoon/sign-language-recognition, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/alpacamoon/sign-language-recognition
- Repository: https://github.com/alpacamoon/sign-language-recognition

## Overview

##### Project Type & Purpose
- This is a local, full-featured Python application built with KivyMD that captures American Sign Language (ASL) via a webcam.
- It processes both static (alphabet and numbers) and dynamic (word gestures) signs and produces text and/or audio outputs.
- In addition, the project incorporates post‐processing for sentence generation (to produce fluent outputs), language translation, and text-to-speech conversion.

##### Analyzing the File Structure
- There are distinct modules/directories that each represent major components of the overall system:
- “kivy_components” and “kivy_main.py”: These support the user interface and camera input using Kivy.
- “action_feature_extraction” and “static_feature_extraction”: These modules contain code for preprocessing video frames and extracting features for dynamic (action-based) signs and static signs respectively.
- “action_recognition” and “static_recognition”: These contain the recognition models (with saved model files and organized keypoints data) that use the extracted features to predict the interpreted sign.
- “sentence_generator”: This module takes recognized individual signs and concatenates them into fluent, human-readable sentences, possibly with additional language models.
- “translation_module”: Handles conversion of the text into other languages using provided JSON language mappings.
- “text_to_speech”: Converts the final text output into audio (e.g., via GTTS or similar libraries).
- “word_segmentation”: Likely helps break or merge recognized characters/words for better contextual accuracy.
- “reference”: Contains supporting files such as sample videos and supplementary CSV/excel reference documents.
- There are also directories for logs, caches, fonts, and requirements which support training, debugging, and dependency management.

##### Components & Their Relationships
- User Interface (UI) Layer:
- Kivy-based UI (“kivy_main.py” and “kivy_components”) handles video capture and user interactions.
- Input (Webcam Capture):
- Captured frames are sent into two parallel feature extraction pipelines:
▪ Dynamic Processing: “action_feature_extraction” extracts features from video sequences of dynamic signs.
▪ Static Processing: “static_feature_extraction” processes static gestures (alphabets/numbers).
- Recognition Engines:
- “action_recognition”: Uses extracted features to classify dynamic ASL gestures.
- “static_recognition”: Recognizes static sign gestures.
- Both engines depend on pre-trained models and produce recognition outputs (labels, probabilities, etc.).
- Natural Language & Post-Processing Components:
- “sentence_generator”: Consolidates single sign outputs into sentences.
- “word_segmentation”: Refines the segmentation of characters or words to enhance accuracy.
- “translation_module”: Maps recognized text into other languages if required.
- Output Layer:
- “text_to_speech”: Converts final text outputs to audio.
- The system may also display the text result visually within the UI.
- Supporting & Logging Components:
- Log directories (within feature extraction modules) record training/testing events for debugging and model improvement.
- Reference assets (videos, CSVs, Excel files) provide ground truth, training samples, and lookup information.

##### Key Architectural Patterns & Design Principles
- Modular / Layered Architecture: Each distinct functionality (e.g., recognition, sentence generation, translation) is encapsulated in its own module. This separation promotes maintainability and clearer data flow.
- Data Abstraction: The inputs and outputs between modules are abstracted (for instance, raw video frames to feature vectors, then to recognized sign classes, and finally to sentence and speech output).
- Use of Pre-trained Models: In the recognition modules, specific models are stored (files in “models/”) to perform classification, showing the machine learning component’s integration.

##### Final Recommendations
"Webcam Input → Feature Extraction → Recognition → Sentence Generation → (Optional Translation) → Text and TTS Output."
- You might also include annotations for any configuration or dependency files (like “requirements.txt”) that hint at external libraries supporting these components.

## Diagram (Mermaid)

```mermaid
graph TD
    %% UI Layer
    subgraph "UI Layer"
        KivyMain("kivy_main.py"):::ui
        KivyComponents("kivy_components"):::ui
    end

    %% Input Acquisition
    WebcamInput("Webcam Input"):::ui

    KivyMain -->|"captures"| WebcamInput
    KivyComponents -->|"renders"| WebcamInput

    %% Feature Extraction Layer
    subgraph "Feature Extraction"
        DynamicFE("Dynamic Feature Extraction"):::processing
        StaticFE("Static Feature Extraction"):::processing
    end

    WebcamInput -->|"sends frames"| DynamicFE
    WebcamInput -->|"sends frames"| StaticFE

    %% Recognition Engines
    subgraph "Recognition Engines"
        ActionRecog("Action Recognition"):::processing
        StaticRecog("Static Recognition"):::processing
    end

    DynamicFE -->|"extracts"| ActionRecog
    StaticFE -->|"extracts"| StaticRecog

    %% Supporting Reference (dotted dependency)
    ActionRecog ---|"refers"| ReferenceData("Reference Data"):::support
    StaticRecog ---|"refers"| ReferenceData

    %% Post-Processing Layer
    subgraph "Post-Processing"
        SentenceGen("Sentence Generator"):::postprocess
        WordSeg("Word Segmentation"):::postprocess
        TranslationMod("Translation Module"):::postprocess
    end

    ActionRecog -->|"output"| SentenceGen
    StaticRecog -->|"output"| SentenceGen
    ActionRecog -->|"refines"| WordSeg
    StaticRecog -->|"refines"| WordSeg

    SentenceGen -->|"merges"| TranslationMod
    WordSeg -->|"merges"| TranslationMod

    %% Output Layer
    subgraph "Output Layer"
        TextDisplay("Text Display"):::output
        TTS("Text-to-Speech"):::output
    end

    TranslationMod -->|"produces"| TextDisplay
    TranslationMod -->|"produces"| TTS

    %% Styles for Supporting Components Subgraph
    subgraph "Supporting Components"
        ReferenceData("Reference Data"):::support
    end

    %% Click Events
    click KivyMain "https://github.com/alpacamoon/sign-language-recognition/blob/main/kivy_main.py"
    click KivyComponents "https://github.com/alpacamoon/sign-language-recognition/tree/main/kivy_components/"
    click DynamicFE "https://github.com/alpacamoon/sign-language-recognition/tree/main/action_feature_extraction/"
    click StaticFE "https://github.com/alpacamoon/sign-language-recognition/tree/main/static_feature_extraction/"
    click ActionRecog "https://github.com/alpacamoon/sign-language-recognition/tree/main/action_recognition/"
    click StaticRecog "https://github.com/alpacamoon/sign-language-recognition/tree/main/static_recognition/"
    click SentenceGen "https://github.com/alpacamoon/sign-language-recognition/tree/main/sentence_generator/"
    click WordSeg "https://github.com/alpacamoon/sign-language-recognition/tree/main/word_segmentation/"
    click TranslationMod "https://github.com/alpacamoon/sign-language-recognition/tree/main/translation_module/"
    click TTS "https://github.com/alpacamoon/sign-language-recognition/tree/main/text_to_speech/"
    click ReferenceData "https://github.com/alpacamoon/sign-language-recognition/tree/main/reference/"

    %% Class Styles
    classDef ui fill:#a6cee3,stroke:#1f78b4,stroke-width:2px;
    classDef processing fill:#b2df8a,stroke:#33a02c,stroke-width:2px;
    classDef postprocess fill:#ffcc99,stroke:#ff8c00,stroke-width:2px;
    classDef output fill:#fdbf6f,stroke:#ff8c00,stroke-width:2px;
    classDef support fill:#dddddd,stroke:#888888,stroke-width:2px,stroke-dasharray: 5 5;
```
