# tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation architecture

> Architecture diagram of the GitHub repository tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation
- Repository: https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation

## Overview

##### Project Type and Purpose
- This project is a deep-learning based real-time recognition system. It uses image capture (likely from a webcam) to detect Indian Sign Language gestures and translates the recognized alphabets into Braille script.
- The project is structured as a research/prototype or experimental ML pipeline built using Jupyter notebooks, model files (Keras models, TensorFlow saved models, and TFLite models), and multiple directories for outputs and results.

##### File Structure to Discover Major Components
- Data Acquisition:
– Notebooks like CapturePicture.ipynb and CollectData.ipynb indicate that images are captured from a live feed or camera.
– The "New pictures" and "Pictures" directories contain image data.
- Data Preparation:
– DatasetPrepare.ipynb is used to prepare the dataset for training or validation.
- Model Training and Benchmarking:
– Multiple notebooks (PrepareModelBenchmark.ipynb, PrepareModelCustom.ipynb) suggest differing approaches to model training (benchmark vs. custom).
– The “models” directory contains pre-trained or reference model files (e.g., actionCO.h5, actionMN.h5, etc.) and weights.
– The presence of numerous saved model directories (saved_models, saved_models_benchmark, etc.) shows that models are being persisted in different formats.
- Inference and Application:
– The app.ipynb notebook is likely used for running real-time inferences and displaying the Braille translation outputs.
- Output and Results:
– Directories like "Output" and "Results" (including accuracy, loss, and architecture PDFs) indicate analysis and visualization phases.
– There is an existing architecture_diagram (both .drawio and .pdf) that could serve as a reference or be updated.
- Export Models for Edge Deployment:
– The tflite_models directory contains TensorFlow Lite models, suggesting a focus on deploying lightweight inference on mobile or embedded devices.

a. Main Components:
– Input Module: Camera/Data Capture Environment
– Data Collection & Storage: Handling new images, dataset directories, and image preprocessing.
– Data Preparation & Preprocessing Module: Using DatasetPrepare.ipynb to normalize and format images.
– Model Training & Evaluation Module:
- Two paths: Benchmark model and Custom model (each with its own training logic and saved weights).
– Model Deployment Module:
- Inference engine in app.ipynb that loads trained models from either “models”, “saved_models”, or the optimized TFLite models.
– Translation Engine:
- A subcomponent that maps the recognized sign language (English alphabet) to its corresponding Braille script.
– Output Display:
- Real-time display through the app, along with saving output PDFs and logs.
b. Relationships and Data Flow:
– The Input Module sends captured images to the Data Preparation Module.
– Preprocessed data is then fed into the Model Training Module for both benchmark and custom models.
– Trained models are saved to “saved_models” directories, and optionally converted to TFLite format.
– In real-time operation, the Inference Engine receives new camera input, processes it using the trained model, and passes the recognized sign to the Translation Engine.
– The system outputs the Braille translation and may update the UI or save logs (in “Output” or “Results” folders).
c. Architectural Patterns and Design Principles:
– Modular design: Each notebook and directory points to a separation of concerns (capture, preprocess, train, infer, output).
– Data pipeline structure: Raw data → preprocessing → model training → inference → translation.
– Experimentation logging: Separate outputs for benchmarks and custom models for performance evaluation.
– Model portability: Use of both Keras/TensorFlow models and TFLite for embedded deployment.
d. Relevant Technologies:
– Python and Jupyter Notebooks for prototyping.
– Deep Learning frameworks (TensorFlow/Keras) suggested by .h5 and saved_model directories.
– TensorFlow Lite for mobile/edge deployment.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Acquisition Subgraph
    subgraph "Data Acquisition"
        CAP1["CapturePicture.ipynb"]:::capture
        CAP2["CollectData.ipynb"]:::capture
        NP["New pictures"]:::storage
        P["Pictures"]:::storage
    end

    %% Preprocessing / Dataset Preparation
    subgraph "Preprocessing"
        DP["DatasetPrepare.ipynb"]:::prep
    end

    %% Model Training & Evaluation Subgraph
    subgraph "Model Training & Evaluation"
        %% Benchmark Training Subgraph
        subgraph "Benchmark Training"
            BM_NB["PrepareModelBenchmark.ipynb"]:::train
            subgraph "Benchmark Model Files"
                BM1["actionCO.h5"]:::model
                BM2["weights_benchmark.h5"]:::model
                BM3["saved_models_benchmark"]:::model
                BM4["model_benchmark.pdf"]:::result
                BM5["accuracy_benchmark.pdf"]:::result
                BM6["loss_benchmark.pdf"]:::result
            end
            BM_NB --> BM1
            BM1 --> BM2
            BM2 --> BM3
            BM3 --> BM4
            BM4 --> BM5
            BM5 --> BM6
        end

        %% Custom Training Subgraph
        subgraph "Custom Training"
            CM_NB["PrepareModelCustom.ipynb"]:::train
            subgraph "Custom Model Files"
                CM1["actionMN.h5"]:::model
                CM2["weights_custom.h5"]:::model
                CM3["saved_models_custom"]:::model
                CM4["model_custom.pdf"]:::result
                CM5["accuracy_custom.pdf"]:::result
                CM6["loss_custom.pdf"]:::result
            end
            CM_NB --> CM1
            CM1 --> CM2
            CM2 --> CM3
            CM3 --> CM4
            CM4 --> CM5
            CM5 --> CM6
        end
    end

    %% Trained Model Artifact (merging benchmark and custom models)
    subgraph "Trained Model Artifact"
        TMA["Trained Model Artifact"]:::trained
    end

    %% Model Conversion for Edge Deployment
    subgraph "Model Conversion (TFLite)"
        TFL["tflite_models"]:::convert
        TFL1["sign.tflite"]:::convert
        TFL2["sign_quant.tflite"]:::convert
        TFL --> TFL1
        TFL --> TFL2
    end

    %% Real-Time Inference & Application
    subgraph "Real-Time Inference"
        RTI["app.ipynb"]:::infer
    end

    %% Braille Translation Module
    subgraph "Braille Translation Module"
        BT["Braille Translation Module"]:::translate
    end

    %% Output & Visualization
    subgraph "Output & Visualization"
        OD["Output Directory"]:::output
        RD["Results Directory"]:::output
        DD["architecture_diagram.drawio"]:::output
        PD["architecture_diagram.pdf"]:::output
    end

    %% Data Flow Connections
    CAP1 --> DP
    CAP2 --> DP
    NP --> DP
    P --> DP

    DP --> BM_NB
    DP --> CM_NB

    BM6 --> TMA
    CM6 --> TMA

    TMA --> RTI
    TMA --- TFL

    RTI --> BT
    BT --> OD
    BT --> RD
    RD --> PD
    RD --> DD

    OD -- "feedback"--> DP

    %% Styles
    classDef capture fill:#f9d5bb,stroke:#d67936,stroke-width:2px
    classDef storage fill:#ffe0b3,stroke:#d67936,stroke-width:2px
    classDef prep fill:#fff2cc,stroke:#d6b656,stroke-width:2px
    classDef train fill:#d5e8d4,stroke:#82b366,stroke-width:2px
    classDef model fill:#dae8fc,stroke:#6c8ebf,stroke-width:2px
    classDef result fill:#e1d5e7,stroke:#9673a6,stroke-width:2px
    classDef trained fill:#ffd6cc,stroke:#ff8c1a,stroke-width:2px
    classDef convert fill:#c9daf8,stroke:#6d9eeb,stroke-width:2px
    classDef infer fill:#fce5cd,stroke:#f1c232,stroke-width:2px
    classDef translate fill:#d9ead3,stroke:#6aa84f,stroke-width:2px
    classDef output fill:#ead1dc,stroke:#b8548e,stroke-width:2px

    %% Click Events
    click CAP1 "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/blob/main/CapturePicture.ipynb"
    click CAP2 "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/blob/main/CollectData.ipynb"
    click NP "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/tree/main/New pictures"
    click P "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/tree/main/Pictures"
    click DP "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/blob/main/DatasetPrepare.ipynb"
    click BM_NB "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/blob/main/PrepareModelBenchmark.ipynb"
    click BM1 "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/blob/main/models/actionCO.h5"
    click BM2 "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/blob/main/models/weights_benchmark.h5"
    click BM3 "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/tree/main/saved_models_benchmark"
    click BM4 "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/blob/main/Results/Architecture/model_benchmark.pdf"
    click BM5 "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/blob/main/Results/Accuracy/accuracy_benchmark.pdf"
    click BM6 "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/blob/main/Results/Loss/loss_benchmark.pdf"
    click CM_NB "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/blob/main/PrepareModelCustom.ipynb"
    click CM1 "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/blob/main/models/actionMN.h5"
    click CM2 "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/blob/main/models/weights_custom.h5"
    click CM3 "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/tree/main/saved_models/saved_models_custom"
    click CM4 "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/blob/main/Results/Architecture/model_custom.pdf"
    click CM5 "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/blob/main/Results/Accuracy/accuracy_custom.pdf"
    click CM6 "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/blob/main/Results/Loss/loss_custom.pdf"
    click RTI "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/blob/main/app.ipynb"
    click TFL "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/tree/main/tflite_models"
    click TFL1 "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/blob/main/tflite_models/sign.tflite"
    click TFL2 "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/blob/main/tflite_models/sign_quant.tflite"
    click OD "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/tree/main/Output"
    click RD "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/tree/main/Results"
    click DD "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/blob/main/architecture_diagram.drawio"
    click PD "https://github.com/tiwarishubham635/real-time-indian-sign-language-recognition-and-braille-script-translation/blob/main/architecture_diagram.pdf"
```
