How asharibkhannn/asl-sign-language-recognition works
Project Overview & Type
• This is a real-time ASL hand sign detection application using deep learning (specifically an LSTM model) and computer vision.
• It is a desktop application with a graphical interface that processes webcam input, performs image preprocessing (such as skin segmentation), and classifies ASL hand signs with a trained model.
• There’s also an integrated performance evaluation tool that computes metrics and visualizations, useful for tuning and benchmarking the system.
Main Components
• User/Environment:
• Webcam: The source of video frames fed into the application.
• User Interface (GUI): Provided by asl_camera_gui.py, which displays the video stream, the confidence meter, and the adjustable threshold.
• Data Preprocessing Module:
• Incorporates hand detection and skin segmentation performed on each captured frame.
• Machine Learning Component:
• ASL Classifier (using the deep learning model in asl_classifier_finetuned.h5).
• Class Mapping: The mapping file (class_mapping.npy) that translates model output indices to ASL letters.
• Launch Scripts:
• Windows: run_asl.bat (for launching the application), run_performance_evaluation.bat.
• Linux/Mac: run_asl.py and run_performance_evaluation.py.
• Performance Evaluation Module:
• Evaluation script(s) (evaluate_model_performance.py if present, or integrated code within run_performance_evaluation.py) that measures and visualizes accuracy, precision, recall, F1 score, inference times, etc.
• Outputs (e.g., reports and visualizations) that might be stored in a generated folder like performance_results.
• Dependencies/External Libraries:
• TensorFlow, OpenCV, NumPy, and Pillow, which underpin the core functionalities.
• The preprocessed image is fed into the LSTM-based classifier model (asl_classifier_finetuned.h5).
• The output is then mapped via the “Class Mapping” (class_mapping.npy) to obtain the corresponding ASL letter.
• Next, connect this result to the “User Interface” block where the recognized sign is visually displayed along with a confidence meter and an adjustable threshold slider.
• Separately, include a branch for “Performance Evaluation”:
• A separate flow when the performance evaluation scripts are run. This flow captures the same or additional data (like inference time, per-class analysis, confusion matrix) and outputs detailed reports.
• This component should connect to a “Results Visualization/Report” block, indicating that the data produced is visualized or stored.
Architectural Patterns and Technologies
• A modular design: front-end (GUI), processing (image preprocessing and model inference), and evaluation (performance metrics). This ensures separation of concerns and easier troubleshooting.
• The data flow is sequential (from image capture, through preprocessing, to inference, and finally display), while performance evaluation runs parallel when specifically invoked.