# tanmayjivnani/sign-language-to-text-and-speech architecture

> Architecture diagram of the GitHub repository tanmayjivnani/sign-language-to-text-and-speech, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tanmayjivnani/sign-language-to-text-and-speech
- Repository: https://github.com/tanmayjivnani/sign-language-to-text-and-speech

## Overview

##### Main Components
- Dataset Collection Module: Represented by the script "collectImgs.py" which is responsible for capturing gesture images from the webcam.
- Data Preprocessing and Feature Extraction Module: This is handled by "createDataset.py" which extracts 42 features per sample (using MediaPipe for hand landmarks) and saves them in a pickle file.
- Model Training Module: Represented by "trainClassifier.py" which takes the preprocessed dataset and trains a Random Forest Classifier. Also note that “model.p” is the pre-trained model file in the repository.
- Inference / Real-Time Gesture Recognition Module: The "main.py" script integrates the previously trained model, processes real-time input from a webcam, stabilizes predictions using a stabilization buffer, and then converts the recognized gestures into words/sentences.
- Text-to-Speech (TTS) and User Interface: Alongside the recognition, a TTS engine (using Pyttsx3) and a Tkinter based GUI display the current alphabet, word, and sentence for user feedback.

##### Relationships and Component Interactions
- Input Flow: Start with the webcam. The image feed is captured in the Dataset Collection stage.
- Preprocessing: Images are then processed using MediaPipe (extracting key landmarks) in the Data Preprocessing component.
- Training: The processed features are fed into a Machine Learning Trainer (using the Random Forest Classifier) which produces a trained model.
- Inference: In real-time, the webcam feed is preprocessed in a similar manner, then the ML model (pre-trained or custom-trained) recognizes the gestures.
- Stabilization & Interpretation: A stabilization buffer smooths out recognition errors and then the recognized gesture is converted into readable text.
- Output: The recognized text is both displayed on the GUI and converted into speech using TTS.

##### Important Architectural Patterns/Design Principles
- Modular Decomposition: Separate scripts for dataset collection, preprocessing, model training, and inference imply a modular design that can be individually maintained or upgraded.
- Data Flow Pipeline: There is clearly a pipeline starting from data acquisition, feature extraction, model building, and finally real-time inference.
- Reusability and Extensibility: The inclusion of training scripts, a pre-trained model, and the potential for future enhancements (dynamic gestures, mobile deployment) underscore the design’s adaptability.
- Real-Time Processing: Emphasis on stabilization and real-time processing is a crucial architectural component to ensure accuracy during inference.

##### Relevant Technologies and Frameworks
- Languages & Libraries: Python, MediaPipe, OpenCV (for image capture and processing), Scikit-learn (for training the Random Forest model).
- GUI: Tkinter is used for building a graphical interface.
- TTS: Pyttsx3 is used for converting text to speech.
- Data Persistence: Use of pickle files for storing processed features and the pre-trained model.

##### Specific Elements
- From Webcam → Dataset Collection → Preprocessing (MediaPipe Feature Extraction)
- From Preprocessed Data → Model Training → Producing A Model (model.p)
- From Webcam (inference loop) → Preprocessing → ML Prediction → Stabilization Buffer
- From Stabilization → Text-Conversion → GUI & TTS
- Adding annotations that reference the supporting libraries or technologies (e.g., a note that "MediaPipe extracts 21 hand landmarks for 42 features").

## Diagram (Mermaid)

```mermaid
flowchart TD
    A("Webcam Feed"):::external

    subgraph "Training Pipeline"
        T1("Dataset Collection (collectImgs.py)"):::training
        T2("Feature Extraction & Preprocessing (createDataset.py) - MediaPipe:42 features"):::training
        T3("Model Training (trainClassifier.py)"):::training
        T4("Pre-trained Model (model.p)"):::training
    end

    subgraph "Real-Time Inference Pipeline"
        I1("Real-Time Inference (main.py)"):::inference
        I2("Stabilization Buffer"):::inference
        I3("GUI & TTS (main.py)"):::inference
    end

    A -->|"captures"| T1
    T1 -->|"sends_images"| T2
    T2 -->|"provides_features"| T3
    T3 -->|"saves_model"| T4
    T4 -->|"loads_model"| I1

    A -->|"live_feed"| I1
    I1 -->|"stabilizes"| I2
    I2 -->|"converts_to_text"| I3

    click T1 "https://github.com/tanmayjivnani/sign-language-to-text-and-speech/blob/main/collectImgs.py"
    click T2 "https://github.com/tanmayjivnani/sign-language-to-text-and-speech/blob/main/createDataset.py"
    click T3 "https://github.com/tanmayjivnani/sign-language-to-text-and-speech/blob/main/trainClassifier.py"
    click T4 "https://github.com/tanmayjivnani/sign-language-to-text-and-speech/blob/main/model.p"
    click I1 "https://github.com/tanmayjivnani/sign-language-to-text-and-speech/blob/main/main.py"
    click I3 "https://github.com/tanmayjivnani/sign-language-to-text-and-speech/blob/main/main.py"

    classDef external fill:#ECECEC,stroke:#000,stroke-width:2px;
    classDef training fill:#FFDD99,stroke:#000,stroke-width:2px;
    classDef inference fill:#B8E1FF,stroke:#000,stroke-width:2px;
```
