How dgovor/sign-language-translator works
• Data Collection Component – represented by data_collection.py and its integration with the MediaPipe Holistic pipeline for capturing hand landmark data.
• Model Training Component – driven by model.py where the neural network (built using a Sequential model with LSTM and Dense layers) is defined, trained, and validated using the collected datasets.
• Real Time Prediction Component – handled by main.py which streams video, processes frames using MediaPipe Holistic, feeds sequences of landmarks to the trained model for sign predictions, and then passes the predicted sentence to the grammar correction module using language_tool_python.
• Utility Functions – represented by my_functions.py that may contain support routines used across the data collection, training, or real time prediction processes.
• Data/Assets – include directories (img) and potentially other assets that support user interfaces for instructions or result displays.
Relationships and interactions
• That data from the Data Collection module is used to build training datasets, which then feed into the Model Training module.
• From Model Training, once the network is trained, its output (the trained model) is used by the Real Time Prediction module to translate input frames into text.
• Integration with MediaPipe Holistic in both Data Collection and Real Time Predictions—these external services are vital to the system’s operation.
Architectural details
• That the project uses a neural network architecture with specific layers: multiple LSTM layers, Dense layers, ReLU activation in hidden layers, and Softmax at the output.
• That data is segmented into sequences (e.g., series of 10 frames) before being sent to the model – this is essential for understanding the temporal dynamics processed by LSTM.
• Mention the use of the Adam optimizer used in model training, and how training-validation splitting (e.g., 90/10 split) is done.
• Design principle of modularity by indicating that the system’s functionalities (collection, training, prediction) are encapsulated within their own scripts.
Relevant technologies and external services
• Python 3.6+
• Java (as required by LanguageTool)
• MediaPipe Holistic for hand tracking
• language_tool_python for grammar correction
• Data visualization components (e.g., the img directory with example gifs that illustrate process steps)