How adityamakkar000/signspeak works
Project Type and Purpose
• The project is primarily a research and open-source tool addressing real-time ASL translation.
• It comprises both hardware interactions (data glove using flex sensors and microcontroller code) and software pipelines (data collection, preprocessing, training experiments, and model implementations using LSTM, GRU, etc.).
Analyzing the File Structure
• Top-Level Structure:
• Files such as .gitignore, LICENSE, and README indicate standard repository practices.
• The requirement file (req.txt) signifies dependency management.
• Key Directories:
• src/dataCollection: Contains the logic for interfacing with the data glove – notably dataCollector.py, glovecode.ino, and stat.py.
• src/experiments: Includes training scripts (LightningTrain.py, RNNTrain.sh, encodertrain.sh) and a dedicated subfolder for experimental data (data/data.csv).
• src/models: Houses different model definitions (GRU.py, LSTM.py, LightningModel.py, encoder.py, generalModels.py), clearly separating the various deep learning architectures.
• src/misc: Contains additional utilities like DataModule.py that may support data manipulation or other supportive functions.
Main Components
a. Data Glove & Sensor Hardware:
• Physical hardware (flex sensors on the glove) along with the microcontroller (Arduino MEGA 2560). This component is responsible for capturing finger movements.
• That this hardware sends real-time sensor data (via serial output) to the system.
b. Data Collection Module (Software Side):
• DataCollection module (src/dataCollection) which includes:
• dataCollector.py – for collecting and processing sensor input.
• glovecode.ino – firmware running on the microcontroller.
• stat.py – for any statistical handling or monitoring.
c. Dataset Storage & Preprocessing:
• Dataset (stored in src/experiments/data/data.csv) that is acquired through the glove.
• A preprocessing step if extra operations are performed before training.
d. Experiments & Training Module:
• Training scripts (LightningTrain.py, RNNTrain.sh, encodertrain.sh) which trigger model training.
• The role of the command-line interface in adjusting hyperparameters.
e. Model Definitions & Inference:
• Out the models defined in src/models: LSTM, GRU, Transformer-based, etc.
• How these modules feed into the training pipeline and eventually into the prediction process.
f. Supporting Modules:
• Src/misc/DataModule.py if it serves to support data handling and integration across components.
Relationships and Data Flow
• From the Physical Data Glove → Arduino (embedded microcontroller code in glovecode.ino) → Data Collection Module (dataCollector.py within the src/dataCollection directory).
• From the Data Collection Module → Dataset Storage (data.csv in src/experiments/data).
• From Dataset Storage → Experiment/Training Scripts (LightningTrain.py, RNNTrain.sh, encodertrain.sh) in src/experiments.
• From Experiment/Training Scripts → Model Modules in src/models, representing training and evaluation.
Architectural Patterns & Design Principles
• Modularity: Separate components (data collection, experimentation, and modeling) are clearly isolated, following a layered approach.
• Pipeline Flow: Emphasizes the sequential nature (from data capture to training) while supporting feedback loops for model optimization.
• Hardware-Software Integration: Demonstrates how low-level sensor data is transformed into high-level model predictions.