How keshx-3/signsense works
This is a desktop AI application (single-machine) for Indian Sign Language (ISL) translation with three main user capabilities:
1. Sign → Text: Webcam video stream → hand/pose feature extraction → CNN/TensorFlow model inference → predicted sign/word → displayed as text (potentially multi-language).
2. Text → Sign: User enters text → maps to sign assets (videos in words/, potentially alphabet assets) → plays visual representation.
3. Chatbot assistance: UI calls an LLM API (README says OpenAI; repo includes gemini.py too) → answers questions/help.
Application entrypoints / orchestration
• AI_Assistant.py (chat assistant / helper UI or assistant logic)
• Sign_Translator.py (translation workflow / pipeline runner)
• read.py, extract.py, k.py (support scripts; likely preprocessing, dataset utilities, experimentation)
UI layer (Frontend in README terms)
• Tkinter GUI (not a folder, but explicitly in README)
• UI as the central hub that triggers:
• webcam capture / sign-to-text flow
• text-to-sign flow
• chatbot calls
• output rendering (text, images, video)
Sign-to-text ML inference assets
• Model files:
• model.h5, action.h5, ISL1.h5
• ISL25/ TensorFlow SavedModel (saved_model.pb, variables/, keras_metadata.pb)
• Libraries (from README): OpenCV, MediaPipe, TensorFlow/Keras, PIL
• Treat these as: “Inference Engine” + “Model Registry/Artifacts” (local filesystem).
Data / media assets for text-to-sign
• words/*.mp4 (word-level sign videos)
• filtered_data/ (training or sample data; includes .webm, .zip)
• sign_translator.ipynb (R&D notebook—document as “offline experimentation”, not runtime)
External AI services
• README: OpenAI API
• File: gemini.py suggests Google Gemini integration or alternate provider
Don’t overthink it: split into many components
For this project, “best and accurate” means you separate:
• Vs extraction vs preprocessing vs inference vs decoding
• UI vs inference vs assets
• local vs external
• runtime vs offline tooling