# facebookresearch/emg2qwerty architecture

> Architecture diagram of the GitHub repository facebookresearch/emg2qwerty, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/emg2qwerty
- Repository: https://github.com/facebookresearch/emg2qwerty

## Overview

##### Main Components
- Data Ingestion and Preprocessing
- Dataset (HDF5 session files and metadata.csv) and the transformation utilities (emg2qwerty/transforms.py and config/transforms/) that reformat or compute spectrogram representations.
- Configuration Management
- Hydra configuration system (with various YAML files in config/, including user-specific configs, model configurations, optimizer configurations, schedulers, decoders, etc.). This is key because it allows users to override parameters (e.g., via command-line overrides for SLURM clusters or custom user settings).
- Model and Training Pipeline
- “emg2qwerty” module’s core components: data handling (data.py), model modules (modules.py), Lightning integrations (lightning.py), and training routines (train.py). Include the checkpointing and model files stored under models/ (both generic and personalized models).
- Decoding & Language Model Integration
- Decoder logic (decoder.py) alongside two decoding strategies (ctc_beam and ctc_greedy found in config/decoder/). Also show dependencies like the 6-gram character-level language model built using KenLM (referenced by the lm scripts, e.g., scripts/lm/).
- External Services and Environment
- Elements such as the setup process (via conda environment and git-lfs for pretrained checkpoints), and a note on the optional use of a Slurm cluster (with the associated cluster configurations in config/cluster/).

##### Relationships and Data Flow
- Preprocessing Pipeline:
- Data to a preprocessing/transform module that converts raw signals into log spectrograms or other formats as required.
- Training Module:
- Model Storage and Checkpointing:
- A component that shows saving and loading model checkpoints stored in the models/ directory (both generic and personalized).
- Decoding/Inference Process:
- Inference phase where either the greedy or beam-search decoder is applied. Highlight the integration with the language model – including a reference to building or loading a 6-gram char LM (from models/lm/).
- Deployment Environment:
- Where external orchestration happens, like launching training either locally, with GPU support, or on a SLURM cluster (via Hydra’s Submitit plugin as indicated in the README).

##### Architectural Patterns and Design Principles
- Modularity: The system is conceptually divided into components such as data handling, preprocessing, modeling/training, and decoding.
- Configurability: The Hydra-based configuration allows users to adjust nearly every aspect of the system (from optimizers to decoders and training environment).
- Extensibility: The design clearly segments generic and personalized user models, suggesting an easy pathway to plug in additional user-specific configurations or modules.
- Reproducible Research: There’s a clear separation between experimental code (under scripts/) and core library code (the emg2qwerty module), with testing infrastructure (unit tests under emg2qwerty/tests/).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        A["Raw Data (HDF5 Sessions & metadata.csv)"]:::data
        B["Data Ingestion (emg2qwerty/data.py)"]:::data
        C["Preprocessing & Transforms"]:::data
        D["Transforms Utility (emg2qwerty/transforms.py)"]:::data
        E["Transform Config (log_spectrogram)"]:::data
    end

    %% Configuration Management
    subgraph "Configuration Management"
        F["Hydra Config Manager"]:::config
        G["Configuration Files (Hydra YAML)"]:::config
    end

    %% Model and Training Pipeline
    subgraph "Model & Training Pipeline"
        H["Training Routine (emg2qwerty/train.py)"]:::training
        I["Lightning Module (emg2qwerty/lightning.py)"]:::training
        J["Model Components (emg2qwerty/modules.py)"]:::training
    end

    %% Model Checkpointing and Storage
    subgraph "Checkpointing & Model Storage"
        K["Generic Checkpoint"]:::checkpoint
        L["Personalized (Finetuned)"]:::checkpoint
        M["Personalized (Random Init)"]:::checkpoint
    end

    %% Decoding & Language Model Integration
    subgraph "Decoding & Language Model Integration"
        N["Decoder Logic (emg2qwerty/decoder.py)"]:::decoding
        O["CTC Beam Config"]:::decoding
        P["CTC Greedy Config"]:::decoding
        Q["Language Model (KenLM)"]:::decoding
        R["LM Scripts"]:::decoding
    end

    %% External Services and Environment
    subgraph "External Services & Environment"
        S["Cluster Deployment"]:::external
        T["Conda Environment Setup"]:::external
    end

    %% Data Flow Connections
    A -->|"ingested"| B
    B -->|"processed"| C
    C --> D
    D --> E

    %% Flow from Data Layer to Training
    E -->|"feeds data"| H

    %% Configuration influences Training and Decoding
    F ---|"provides settings"| H
    F ---|"provides settings"| N
    G ---|"includes"| F

    %% Training Pipeline internal flow
    H --> I
    I --> J
    J -->|"updates"| K
    J -->|"updates"| L
    J -->|"updates"| M

    %% Decoding & Language Model Flow
    H -->|"outputs for inference"| N
    N --> O
    N --> P
    O --> Q
    P --> Q
    Q --> R

    %% External Services Connections
    S ---|"cluster configs"| G
    T ---|"env setup"| H

    %% Click Events
    click B "https://github.com/facebookresearch/emg2qwerty/blob/main/emg2qwerty/data.py"
    click D "https://github.com/facebookresearch/emg2qwerty/blob/main/emg2qwerty/transforms.py"
    click E "https://github.com/facebookresearch/emg2qwerty/blob/main/config/transforms/log_spectrogram.yaml"
    click G "https://github.com/facebookresearch/emg2qwerty/tree/main/config/"
    click H "https://github.com/facebookresearch/emg2qwerty/blob/main/emg2qwerty/train.py"
    click I "https://github.com/facebookresearch/emg2qwerty/blob/main/emg2qwerty/lightning.py"
    click J "https://github.com/facebookresearch/emg2qwerty/blob/main/emg2qwerty/modules.py"
    click K "https://github.com/facebookresearch/emg2qwerty/blob/main/models/generic.ckpt"
    click L "https://github.com/facebookresearch/emg2qwerty/tree/main/models/personalized-finetuned/"
    click M "https://github.com/facebookresearch/emg2qwerty/tree/main/models/personalized-randominit/"
    click N "https://github.com/facebookresearch/emg2qwerty/blob/main/emg2qwerty/decoder.py"
    click O "https://github.com/facebookresearch/emg2qwerty/blob/main/config/decoder/ctc_beam.yaml"
    click P "https://github.com/facebookresearch/emg2qwerty/blob/main/config/decoder/ctc_greedy.yaml"
    click Q "https://github.com/facebookresearch/emg2qwerty/tree/main/models/lm/"
    click R "https://github.com/facebookresearch/emg2qwerty/tree/main/scripts/lm/"
    click S "https://github.com/facebookresearch/emg2qwerty/tree/main/config/cluster/"
    click T "https://github.com/facebookresearch/emg2qwerty/blob/main/environment.yml"

    %% Styles
    classDef data fill:#ADD8E6,stroke:#00008B,stroke-width:2px;
    classDef config fill:#90EE90,stroke:#006400,stroke-width:2px;
    classDef training fill:#FFD700,stroke:#B8860B,stroke-width:2px;
    classDef checkpoint fill:#DDA0DD,stroke:#8B008B,stroke-width:2px;
    classDef decoding fill:#AFEEEE,stroke:#008B8B,stroke-width:2px;
    classDef external fill:#F08080,stroke:#8B0000,stroke-width:2px;
```
