# keshx-3/signsense architecture

> Architecture diagram of the GitHub repository keshx-3/signsense, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/keshx-3/signsense
- Repository: https://github.com/keshx-3/signsense

## Overview

This is a **desktop AI application** (single-machine) for **Indian Sign Language (ISL) translation** with three main user capabilities:

1. **Sign → Text**: Webcam video stream → hand/pose feature extraction → CNN/TensorFlow model inference → predicted sign/word → displayed as text (potentially multi-language).
2. **Text → Sign**: User enters text → maps to sign assets (videos in `words/`, potentially alphabet assets) → plays visual representation.
3. **Chatbot assistance**: UI calls an LLM API (README says OpenAI; repo includes `gemini.py` too) → answers questions/help.

##### Application entrypoints / orchestration
- `AI_Assistant.py` (chat assistant / helper UI or assistant logic)
- `Sign_Translator.py` (translation workflow / pipeline runner)
- `read.py`, `extract.py`, `k.py` (support scripts; likely preprocessing, dataset utilities, experimentation)

##### UI layer (Frontend in README terms)
- Tkinter GUI (not a folder, but explicitly in README)
- UI as the **central hub** that triggers:
- webcam capture / sign-to-text flow
- text-to-sign flow
- chatbot calls
- output rendering (text, images, video)

##### Sign-to-text ML inference assets
- Model files:
- `model.h5`, `action.h5`, `ISL1.h5`
- `ISL25/` TensorFlow SavedModel (`saved_model.pb`, `variables/`, `keras_metadata.pb`)
- Libraries (from README): OpenCV, MediaPipe, TensorFlow/Keras, PIL
- Treat these as: **“Inference Engine”** + **“Model Registry/Artifacts”** (local filesystem).

##### Data / media assets for text-to-sign
- `words/*.mp4` (word-level sign videos)
- `filtered_data/` (training or sample data; includes `.webm`, `.zip`)
- `sign_translator.ipynb` (R&D notebook—document as “offline experimentation”, not runtime)

##### External AI services
- README: OpenAI API
- File: `gemini.py` suggests Google Gemini integration or alternate provider

##### Don’t overthink it: split into many components
For this project, “best and accurate” means you **separate**:
- Vs extraction vs preprocessing vs inference vs decoding
- UI vs inference vs assets
- local vs external
- runtime vs offline tooling

## Diagram (Mermaid)

```mermaid
flowchart TD

%% =========================
%% Diagram A — Context / Container-level
%% =========================
subgraph "Diagram A — Context (High-level)"
direction TB

U["User"]:::external

subgraph "Local Machine (Single-process Desktop App)"
direction TB

APP["Desktop App"]:::app
UI["Tkinter GUI"]:::ui

subgraph "Local Runtime Dependencies"
direction TB
MODELS[("Model Artifacts\n*.h5 + SavedModel/")]:::model
ASSETS[("Media Assets\nwords/*.mp4 + alphabet/")]:::asset
end

subgraph "Local ML Inference (No Server)"
direction TB
INFER["TensorFlow/Keras Inference Engine"]:::ml
end

subgraph "Chatbot Client"
direction TB
LLMCLIENT["LLM API Client\n(OpenAI / Gemini)"]:::service
end

end

LLMAPI["External LLM Provider API\n(OpenAI / Gemini)"]:::externalSvc

U <-->|"interacts"| UI
UI -->|"orchestrates"| APP
APP -->|"loads"| MODELS
APP -->|"reads"| ASSETS
APP -->|"invokes"| INFER
UI -->|"prompt/response"| LLMCLIENT
LLMCLIENT -->|"HTTPS (prompt)"| LLMAPI
LLMAPI -->|"HTTPS (answer)"| LLMCLIENT
end

%% =========================
%% Diagram B — Runtime Pipelines (Sign↔Text + Chat)
%% =========================
subgraph "Diagram B — Runtime Pipelines (Most important)"
direction TB

subgraph "Flow 1 — Sign → Text (Real-time)"
direction TB
F1_START["Start Capture (UI action)"]:::ui
CAM["Webcam Capture (OpenCV)"]:::pipe
FRAMES["Video Frames\n(frames RGB)"]:::data
MP["MediaPipe Landmark Extractor"]:::pipe
LM["Hand/Pose Landmarks\n(x,y,z)"]:::data
PRE["Preprocessing\n(resize/normalize/sequence buffer)"]:::pipe
TENSOR["Model Input Tensor\n(NxHxWxC)"]:::data
TF["CNN/TensorFlow Inference"]:::ml
PROBS["Class Probabilities"]:::data
POST["Postprocessing\n(threshold + smoothing + label map)"]:::pipe
TOKEN["Predicted Sign Token"]:::data
UI_TEXT["UI Text Renderer\n(+ optional language mapping)"]:::ui

F1_START -->|"start"| CAM
CAM -->|"frames (RGB)"| FRAMES
FRAMES -->|"extract"| MP
MP -->|"landmarks"| LM
LM -->|"prepare"| PRE
PRE -->|"tensor"| TENSOR
TENSOR -->|"infer"| TF
TF -->|"class probs"| PROBS
PROBS -->|"decode"| POST
POST -->|"token"| TOKEN
TOKEN -->|"text"| UI_TEXT

MODELSTORE[("Model Files on Disk\nmodel.h5 / action.h5 / ISL1.h5 / ISL25/")]:::model
TF <-->|"load model"| MODELSTORE
end

subgraph "Flow 2 — Text → Sign (Asset Playback)"
direction TB
F2_START["Enter Text (UI action)"]:::ui
TEXT["Input Text"]:::data
MAP["Tokenizer/Mapper\n(word-level then fallback to alphabet)"]:::pipe
TOKENS["Mapped Tokens\n(words/letters)"]:::data
LOAD["Asset Loader"]:::pipe
WORDLIB[("Word Video Library\nwords/*.mp4")]:::asset
ALPHALIB[("Alphabet Asset Library\nalphabet/*")]:::asset
PLAY["Video/Image Playback\n(embedded in UI)"]:::ui

F2_START -->|"text"| TEXT
TEXT -->|"tokenize"| MAP
MAP -->|"tokens"| TOKENS
TOKENS -->|"resolve assets"| LOAD
LOAD -->|"word exists?"| WORDLIB
LOAD -->|"else spell letters"| ALPHALIB
WORDLIB -->|"mp4 stream"| PLAY
ALPHALIB -->|"image/video stream"| PLAY
end

subgraph "Flow 3 — Chatbot Assistance"
direction TB
F3_START["Ask Question (UI action)"]:::ui
PROMPT["Prompt"]:::data
CHATMOD["AI_Assistant / Provider Adapter"]:::service
EXTLLM["External LLM API\n(OpenAI / Gemini)"]:::externalSvc
ANSWER["Answer Text"]:::data
CHATUI["UI Chat Display"]:::ui

F3_START -->|"prompt"| PROMPT
PROMPT -->|"send"| CHATMOD
CHATMOD -->|"HTTPS request"| EXTLLM
EXTLLM -->|"HTTPS response"| CHATMOD
CHATMOD -->|"answer"| ANSWER
ANSWER -->|"render"| CHATUI
end

NOTE["Runtime Note:\nUI thread + background worker(s)\nfor capture/inference"]:::note
end

%% =========================
%% Diagram C — Module / Repo Mapping
%% =========================
subgraph "Diagram C — Modules mapped to Repo"
direction TB

subgraph "Application Layer"
direction TB
MAIN["main.py\n(App Bootstrap / Composition Root)"]:::app
GUI["Tkinter GUI\n(Presentation layer)"]:::ui
MAIN -->|"starts"| GUI
end

subgraph "Core Workflows"
direction TB
ST["Sign_Translator.py\n(Orchestrator: Sign→Text + Text→Sign)"]:::service
GUI -->|"calls"| ST

subgraph "Sign→Text Pipeline Modules"
direction TB
CVCAP["OpenCV Capture"]:::pipe
MPEX["MediaPipe Extractor"]:::pipe
PP["Preprocess + Sequence Buffer"]:::pipe
INF["TF/Keras Inference"]:::ml
DEC["Decode/Postprocess\n(smoothing + label map)"]:::pipe
ST -->|"uses"| CVCAP
CVCAP -->|"frames"| MPEX
MPEX -->|"landmarks"| PP
PP -->|"tensor"| INF
INF -->|"probs"| DEC
DEC -->|"token/text"| GUI
end

subgraph "Text→Sign Modules"
direction TB
TMAP["Tokenizer/Mapper\n(word→video else alphabet)"]:::pipe
ALOAD["Asset Loader"]:::pipe
ST -->|"uses"| TMAP
TMAP -->|"token list"| ALOAD
ALOAD -->|"playback"| GUI
end
end

subgraph "Chatbot Modules"
direction TB
ASSIST["AI_Assistant.py\n(Chat assistant logic)"]:::service
GEM["gemini.py\n(Gemini provider client)"]:::service
LLMAPI2["External LLM Provider API\n(OpenAI / Gemini)"]:::externalSvc
GUI -->|"prompt"| ASSIST
ASSIST -->|"provider adapter"| GEM
ASSIST -->|"HTTPS"| LLMAPI2
GEM -->|"HTTPS"| LLMAPI2
LLMAPI2 -->|"answer"| ASSIST
ASSIST -->|"render"| GUI
end

subgraph "Local Artifacts (Runtime Dependencies)"
direction TB
H5A[("model.h5")]:::model
H5B[("action.h5")]:::model
H5C[("ISL1.h5")]:::model
SM[("ISL25/ (SavedModel)")]:::model
SMPB["saved_model.pb"]:::artifact
SMV[("variables/")]:::artifact
WORDDIR[("words/")]:::asset
ALPHADIR[("alphabet/")]:::asset

INF -->|"loads"| H5A
INF -->|"loads"| H5B
INF -->|"loads"| H5C
INF -->|"loads"| SM
SM -->|"contains"| SMPB
SM -->|"contains"| SMV
ALOAD -->|"reads"| WORDDIR
ALOAD -->|"reads"| ALPHADIR
end

subgraph "Offline / Dev Tools (Not runtime path)"
direction TB
FD[("filtered_data/")]:::offline
NB["sign_translator.ipynb"]:::offline
EX["extract.py"]:::offline
RD["read.py"]:::offline
KPY["k.py"]:::offline

EX -->|"generates/filters"| FD
NB -->|"experiments with"| FD
RD -->|"utility reads"| FD
KPY -->|"experimentation"| FD
end
end

%% =========================
%% Click Events (from component_mapping)
%% =========================
click MAIN "https://github.com/keshx-3/signsense/blob/main/main.py"
click ST "https://github.com/keshx-3/signsense/blob/main/Sign_Translator.py"
click ASSIST "https://github.com/keshx-3/signsense/blob/main/AI_Assistant.py"
click GEM "https://github.com/keshx-3/signsense/blob/main/gemini.py"
click H5A "https://github.com/keshx-3/signsense/blob/main/model.h5"
click H5B "https://github.com/keshx-3/signsense/blob/main/action.h5"
click H5C "https://github.com/keshx-3/signsense/blob/main/ISL1.h5"
click SM "https://github.com/keshx-3/signsense/tree/main/ISL25/"
click SMPB "https://github.com/keshx-3/signsense/blob/main/ISL25/saved_model.pb"
click SMV "https://github.com/keshx-3/signsense/tree/main/ISL25/variables/"
click WORDDIR "https://github.com/keshx-3/signsense/tree/main/words/"
click ALPHADIR "https://github.com/keshx-3/signsense/tree/main/alphabet/"
click FD "https://github.com/keshx-3/signsense/tree/main/filtered_data/"
click NB "https://github.com/keshx-3/signsense/blob/main/sign_translator.ipynb"
click EX "https://github.com/keshx-3/signsense/blob/main/extract.py"
click RD "https://github.com/keshx-3/signsense/blob/main/read.py"
click KPY "https://github.com/keshx-3/signsense/blob/main/k.py"

%% Extra mapped assets (examples)
YES["words/Yes.mp4"]:::assetFile
NO["words/No.mp4"]:::assetFile
COR["words/Correct.mp4"]:::assetFile
WORDDIR -->|"contains"| YES
WORDDIR -->|"contains"| NO
WORDDIR -->|"contains"| COR
click YES "https://github.com/keshx-3/signsense/blob/main/words/Yes.mp4"
click NO "https://github.com/keshx-3/signsense/blob/main/words/No.mp4"
click COR "https://github.com/keshx-3/signsense/blob/main/words/Correct.mp4"

%% =========================
%% Styles
%% =========================
classDef ui fill:#1e88e5,stroke:#0d47a1,color:#ffffff,stroke-width:1px
classDef app fill:#1565c0,stroke:#0d47a1,color:#ffffff,stroke-width:1px
classDef pipe fill:#90a4ae,stroke:#455a64,color:#111111,stroke-width:1px
classDef ml fill:#7e57c2,stroke:#4a148c,color:#ffffff,stroke-width:1px
classDef model fill:#8e24aa,stroke:#4a148c,color:#ffffff,stroke-width:1px
classDef asset fill:#43a047,stroke:#1b5e20,color:#ffffff,stroke-width:1px
classDef assetFile fill:#66bb6a,stroke:#1b5e20,color:#0b2a0f,stroke-width:1px
classDef service fill:#546e7a,stroke:#263238,color:#ffffff,stroke-width:1px
classDef external fill:#f5f5f5,stroke:#616161,color:#111111,stroke-width:1px
classDef externalSvc fill:#fb8c00,stroke:#e65100,color:#111111,stroke-width:1px
classDef data fill:#e0e0e0,stroke:#616161,color:#111111,stroke-width:1px
classDef offline fill:#c8e6c9,stroke:#2e7d32,color:#111111,stroke-width:1px
classDef artifact fill:#d1c4e9,stroke:#4a148c,color:#111111,stroke-width:1px
classDef note fill:#fff3e0,stroke:#ef6c00,color:#111111,stroke-width:1px,stroke-dasharray: 4 3
```
