# dev-d-gr8/signsense architecture

> Architecture diagram of the GitHub repository dev-d-gr8/signsense, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dev-d-gr8/signsense
- Repository: https://github.com/dev-d-gr8/signsense

## Overview

##### Project Type and Purpose
- This is a machine-learning pipeline for real-time ASL recognition and sentence construction. It has two main modes:
- Training mode – building a transformer model from prerecorded data
- Inference mode – live video capture → gesture recognition → sentence generation

##### File Structure
- Top-level files:
- 10042041.parquet – tabular dataset for training
- ASL_Model_Training.ipynb – Jupyter notebook orchestrating the training pipeline
- app.py – main script for live inference (video capture and translation)
- requirements.txt – Python dependencies
- README.md, LICENSE.md – docs
- No separate “src/” or “frontend/” module—both training and serving live in single repo.

##### Extract Architecture Insights from README
- Core libraries:
- OpenCV & MediaPipe: real-time video capture and hand landmark detection
- TensorFlow: building/testing the transformer model
- Gemini-Pro LLM API: constructs full sentences from recognized ASL tokens
- Real-time pipeline: video → preprocessing → inference → LLM → text output

##### Main Components & Their Interactions
a. Training Subsystem
- Data Loader (reads 10042041.parquet)
- Preprocessing (data augmentation / MediaPipe offline)
- Model Trainer (Transformer architecture in TensorFlow)
- Model Artifacts (saved weights)
b. Inference Subsystem
- Video Capture Module (OpenCV)
- Preprocessing Module (MediaPipe hand/pose extraction)
- Inference Engine (loads trained Transformer weights + TensorFlow)
- Sentence Constructor (Gemini-Pro LLM API)
- Output Renderer (console/window overlay)
c. External Dependencies
- Gemini-Pro LLM API (external service)
- Python environment (managed via requirements.txt)

##### Relationships & Data Flows
- Training Flow: Dataset → Notebook → Model Artifacts
- Inference Flow: Camera → Preprocessor → Model Inference → LLM API → UI

##### Architectural Patterns & Principles
- Pipeline Pattern: clearly separates capture, preprocess, inference, postprocess
- Modular Components: each major step (capture, preprocessing, inference, API call) is a distinct block
- Single Process Monolith: no microservices; all live functionality runs in app.py
- Offline vs. Online Modes: training “batch” vs. inference “streaming”

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Training Pipeline
    subgraph "Training Pipeline"
        DL[("(Data Loader)\n10042041.parquet")]:::storage
        Notebook["(Training Notebook)\nASL_Model_Training.ipynb"]:::local
        PreprocOffline["(Preprocessing)\nMediaPipe Offline"]:::local
        Trainer["(Model Trainer)\nTransformer (TensorFlow)"]:::local
        ModelArtifacts[("(Model Artifacts)\nSaved Weights")]:::storage
    end

    %% Serving Pipeline
    subgraph "Serving Pipeline"
        VideoCap["(Video Capture)\nOpenCV"]:::local
        Preproc["(Preprocessing)\nMediaPipe Real-Time"]:::local
        Inference["(Inference Engine)\nTransformer Model (TensorFlow)"]:::local
        LLM["(Sentence Constructor)\nGemini-Pro LLM API"]:::external
        Output["(Output Renderer)\nConsole/Overlay"]:::local
    end

    %% Docs & Environment
    subgraph "Documentation & Config"
        Req[">Requirements\nrequirements.txt<"]:::env
        README["README.md"]:::docs
        LICENSE["LICENSE.md"]:::docs
    end

    %% Training Flow
    DL --> Notebook
    Notebook --> PreprocOffline
    PreprocOffline --> Trainer
    Trainer --> ModelArtifacts

    %% Serving Flow
    VideoCap --> Preproc
    Preproc --> Inference
    Inference --> LLM
    LLM --> Output

    %% Model Artifact usage
    ModelArtifacts --> Inference

    %% Click Events
    click DL "https://github.com/dev-d-gr8/signsense/blob/main/10042041.parquet"
    click Notebook "https://github.com/dev-d-gr8/signsense/blob/main/ASL_Model_Training.ipynb"
    click VideoCap "https://github.com/dev-d-gr8/signsense/blob/main/app.py"
    click Preproc "https://github.com/dev-d-gr8/signsense/blob/main/app.py"
    click Inference "https://github.com/dev-d-gr8/signsense/blob/main/app.py"
    click LLM "https://github.com/dev-d-gr8/signsense/blob/main/app.py"
    click Output "https://github.com/dev-d-gr8/signsense/blob/main/app.py"
    click Req "https://github.com/dev-d-gr8/signsense/blob/main/requirements.txt"
    click README "https://github.com/dev-d-gr8/signsense/blob/main/README.md"
    click LICENSE "https://github.com/dev-d-gr8/signsense/blob/main/LICENSE.md"

    %% Styles
    classDef local fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef storage fill:#90EE90,stroke:#000,stroke-width:1px
    classDef external fill:#FFA500,stroke:#000,stroke-width:1px
    classDef env fill:#D3D3D3,stroke:#000,stroke-width:1px
    classDef docs fill:#E0E0E0,stroke:#000,stroke-width:1px
```
