How pranaav003/aslwebsite works
Project Type & Purpose
• ASLWebApp is a full-stack, browser-based ASL translation and summarization application.
• It uses a React SPA for real-time camera capture and landmark detection, a Flask API backend for serving static assets, hosting ML models (TensorFlow/Keras & LSTM), and orchestrating calls to OpenAI’s GPT-3.5 Turbo.
File Structure Analysis
• Top-level directory:
– README.md, .DS_Store
– /backend (contains Flask app, models, frontend code, tests, templates)
• backend/app.py
– Entry point for the Flask server, defines REST endpoints.
• backend/frontend
– A React application (package.json, src/, public/)
– Built via npm; output served as static files by Flask.
• backend/asl_lstm_model2.h5, final_model.keras, label_encoder2.pkl
– Pretrained ML artifacts loaded at startup.
• backend/templates/index.html
– Flask template to mount the React build.
• backend/.env.example & requirements.txt
– Environment variables (OPENAI_API_KEY, CAMERA_INDEX, PORT) & Python deps.
• Tests (test_5.py, test_finger.py, test_gpt.py)
– Unit tests for core functionality.
README Insights
• Real-time gesture recognition (hello, thanks, I love you) runs “in your browser via a Flask API.”
• MediaPipe JS extracts landmarks in-browser; Python models perform inference.
• Every 10 s, Flask backend batches recognized gestures and invokes OpenAI API for a context summary.
• Finger-spelling LSTM decodes 26 letters; hold-to-lock behavior in UI.
Main Components & Interactions
A. Client (Browser + React UI)
• Captures camera feed → runs MediaPipe JS to extract hand landmarks.
• Sends POST /predict-gesture or POST /predict-fingerspelling to Flask with landmark payload.
• Displays live subtitles & finger-spell text; requests summaries every 10 s.
B. Flask Backend (app.py)
• Serves React static build (templates/index.html + /static).
• Loads ML models (TensorFlow/Keras & LSTM + label encoder) on startup.
• Exposes REST endpoints:
• /predict-gesture → returns one of {hello, thanks, I love you}
• /predict-fingerspelling → returns letters sequence
• /summarize → calls OpenAI API with recent gestures, returns English summary
C. ML Artifacts
• final_model.keras for 3-class gestures
• asl_lstm_model2.h5 + label_encoder2.pkl for finger spelling
D. External Service
• OpenAI GPT-3.5 Turbo API for natural-language summarization
E. Environment / Config
• .env (OPENAI_API_KEY, CAMERA_INDEX, PORT)
• requirements.txt (Flask, TensorFlow/Keras, python-dotenv, openai)
• frontend package.json (React, MediaPipe, axios or fetch)
Architectural Patterns & Principles
• Separation of concerns: React UI handles capture and landmark detection; Flask handles ML inference & external API calls.
• Stateless REST APIs: each request carries payload; no persistent session.
• Single deployment unit: backend contains both API and static serve.
• Model loading at startup avoids repeated I/O.
Final Checklist
• Clear component labels (Browser/React, Flask API, ML Models, OpenAI)
• Callouts for key files/dirs: /backend/app.py, /backend/frontend/build, /backend/*.h5 / .keras, .env
• Architecture pattern note: “Stateless REST + ML model hosting”