How facebookresearch/emg2qwerty works
Main Components
• Data Ingestion and Preprocessing
• Dataset (HDF5 session files and metadata.csv) and the transformation utilities (emg2qwerty/transforms.py and config/transforms/) that reformat or compute spectrogram representations.
• Configuration Management
• Hydra configuration system (with various YAML files in config/, including user-specific configs, model configurations, optimizer configurations, schedulers, decoders, etc.). This is key because it allows users to override parameters (e.g., via command-line overrides for SLURM clusters or custom user settings).
• Model and Training Pipeline
• “emg2qwerty” module’s core components: data handling (data.py), model modules (modules.py), Lightning integrations (lightning.py), and training routines (train.py). Include the checkpointing and model files stored under models/ (both generic and personalized models).
• Decoding & Language Model Integration
• Decoder logic (decoder.py) alongside two decoding strategies (ctc_beam and ctc_greedy found in config/decoder/). Also show dependencies like the 6-gram character-level language model built using KenLM (referenced by the lm scripts, e.g., scripts/lm/).
• External Services and Environment
• Elements such as the setup process (via conda environment and git-lfs for pretrained checkpoints), and a note on the optional use of a Slurm cluster (with the associated cluster configurations in config/cluster/).
Relationships and Data Flow
• Preprocessing Pipeline:
• Data to a preprocessing/transform module that converts raw signals into log spectrograms or other formats as required.
• Training Module:
• Model Storage and Checkpointing:
• A component that shows saving and loading model checkpoints stored in the models/ directory (both generic and personalized).
• Decoding/Inference Process:
• Inference phase where either the greedy or beam-search decoder is applied. Highlight the integration with the language model – including a reference to building or loading a 6-gram char LM (from models/lm/).
• Deployment Environment:
• Where external orchestration happens, like launching training either locally, with GPU support, or on a SLURM cluster (via Hydra’s Submitit plugin as indicated in the README).
Architectural Patterns and Design Principles
• Modularity: The system is conceptually divided into components such as data handling, preprocessing, modeling/training, and decoding.
• Configurability: The Hydra-based configuration allows users to adjust nearly every aspect of the system (from optimizers to decoders and training environment).
• Extensibility: The design clearly segments generic and personalized user models, suggesting an easy pathway to plug in additional user-specific configurations or modules.
• Reproducible Research: There’s a clear separation between experimental code (under scripts/) and core library code (the emg2qwerty module), with testing infrastructure (unit tests under emg2qwerty/tests/).