How bacemkarray/asl-to-english-translator works
Project type and purpose
• This is a full-stack AI application for real-time ASL-to-English translation.
• The “frontend/acernity_ui” folder houses a Next.js-based UI; “modelTrainingAndPrediction” contains Python scripts and TensorFlow/Keras models for both training and live prediction; “opencvdownloads” is an experimental sandbox for OpenCV-based frame capture and processing.
File structure
• Top-level:
• frontend/acernity_ui: Next.js app (React, Tailwind CSS)
• modelTrainingAndPrediction: Python ML pipeline (frame capture, predictionHandler, .h5 classifiers, Jupyter notebooks)
• opencvdownloads: proof-of-concept scripts for raw OpenCV frame handling
• Configuration files (.eslintrc, tsconfig, postcss, tailwind) indicate a standard React/Next.js build and styling pipeline.
README for additional insights
• Describes real-time ASL gesture capture → English text conversion
• Lists Python, OpenCV, TensorFlow/Keras, Streamlit (note: the file tree doesn’t contain a Streamlit UI—likely an earlier iteration)
• Workflow: video frames in → ML model predicts sign → text displayed in UI
a. Main components:
1) UI Layer (Next.js “acernity_ui”)
2) Prediction Service (Python microservice hosting frameAndPredictionHandling.py and predictionHandler.py, exposing a REST/WebSocket API)
3) ML Model Store (TensorFlow/Keras .h5 files)
4) Offline Training Pipeline (Jupyter notebooks & helper scripts produce new .h5 models)
5) Experimental Sandbox (OpenCV scripts under opencvdownloads) – optional, separate from production
6) External Inputs/Outputs (User’s Webcam, Browser, Optional Database or Logging service)
b. Relationships & interactions:
• The Next.js UI captures webcam frames via getUserMedia → batches frames → sends HTTP/WebSocket requests to the Prediction Service.
• The Prediction Service loads the .h5 model from ML Model Store → runs inference → returns JSON { sign: “A”, confidence: 0.98 }.
• UI renders translated text in real time.
• Offline Training Pipeline reads labeled datasets → trains new models → writes updated .h5 files back to ML Model Store.
• (Optional) Experimental Sandbox can feed new data or test alternate pipelines but is isolated from production inference.
c. Architectural patterns & principles:
• Microservice separation: UI vs. ML service
• Model versioning: store multiple .h5 files, swap by naming convention or endpoint parameter
• Stateless prediction API for scalability
• Offline/online split: heavy training jobs do not interfere with low-latency inference
d. Key technologies & frameworks:
• Frontend: Next.js, React, Tailwind CSS, TypeScript
• Backend: Python (FastAPI/Flask), OpenCV, TensorFlow/Keras
• Data exchange: REST or WebSocket, JSON payloads
• Packaging/build: npm, Python venv/requirements.txt