How oriartyz/signactioninterpreter works
Main Components
• Input / Data Acquisition Module: This includes the real-time video feed (from a camera or video file) and the MediaPipe Holistic tool for extracting keypoints.
• Data Preprocessing and Storage: Once keypoints are extracted, the data (saved as .npy files in various subdirectories inside MP_Data labeled “hello”, “iloveyou”, “thanks”, etc.) is preprocessed. The subfolder structure indicates labeled datasets for different sign actions.
• Logging and Monitoring: The Logs directory shows TensorBoard event files, which indicates that training progress, metrics, and visualizations are being logged.
• Model Training Module: This is represented in notebooks such as Action Detection.ipynb. Here, the pipeline includes loading the preprocessed keypoints, building the LSTM-based model (as specified in the README), training it, evaluating performance, and finally saving the trained model (action.h5).
• Model Inference / Real-time Prediction Module: Once the model is trained and saved, another component (also likely implemented in notebooks and scripts) takes live video, extracts keypoints with MediaPipe, processes them in the same way as training data, and feeds them to the trained LSTM model to predict the corresponding sign language action.
• Auxiliary Resources: The “ressource” directory (which includes an image used in the README) can be shown as supplementary material that aids in documentation and UI design for the project.
Relationships and Interactions
• Data flows from the Input Module to the Preprocessing Module where keypoints are extracted. Preprocessed data is then stored (shown as MP_Data) and used to train the model.
• The training process generates logs (in the Logs directory) and ultimately produces the saved model file (action.h5).
• During real-time predictions, the live data is processed through the same extraction and preprocessing pipeline and then sent to the model for interpretation.
• There is a feedback loop where the performance (e.g., evaluation runs) can trigger further retraining or fine-tuning of the model.
Important Architectural Patterns and Design Principles
• Modularity: Each part (data extraction, preprocessing, model training, and inference) is clearly separated, which helps maintain and upgrade the system.
• Pipeline Design: Data flows through distinct stages. This pattern resembles a typical machine learning pipeline with offline training and online prediction paths.
• Reproducibility and Logging: The use of TensorBoard logs emphasizes the importance of monitoring and reproducibility.
• Use of Notebooks: The code is organized as notebooks (with checkpoints) that make prototyping and visualization integral to development.
Relevant Technologies
• TensorFlow and Keras: For building, training, and saving the LSTM model.
• MediaPipe: For keypoint extraction from video.
• OpenCV: For handling video capture and processing.
• Matplotlib: For plotting and visualizing results during training or evaluation.
• Python Notebooks: Serve as a development and experimentation environment.