How dslaborg/sumo works
Main Components
• Data Input & Preprocessing:
• Input: EEG data (e.g., “input/eeg-sample.npy” and pickled subject data generated via create_data_splits.py).
• Preprocessing: Embedded in the prediction script (predict_plain_data.py) as well as within the training pipeline (downsampling, Butterworth filtering, and z-transformation).
• Configuration Management:
• YAML Files in “config/” (e.g. default.yaml, final.yaml, predict.yaml) supply experiment parameters such as model architecture, training specifics, and evaluation metrics.
• These configuration settings dynamically affect data loading, model parameters, loss functions, and training regimes.
• Training & Evaluation:
• Core driver scripts in “bin/”: train.py and eval.py which use the PyTorch Lightning framework.
• These scripts orchestrate the training process, logging (console and Tensorboard), early stopping, and performance evaluation using configurations and data splits.
• Model & Core Library:
• The “sumo/” package is the heart of the project and is divided into several modules:
• sumo/model: Contains the SUMO U-Net architecture and model partitioning.
• sumo/data: Responsible for data loaders and any data manipulation functions.
• sumo/loss: Implements the loss functions (e.g., GeneralizedDiceLoss) used during training.
• sumo/evaluation: Includes metrics and performance analysis utilities.
• sumo/log: Handles logging mechanisms across training and evaluation.
• sumo/config: Wraps configuration loading or custom parameter handling.
• Auxiliary & Analysis Scripts:
• Scripts in the “scripts/” directory include implementations for the A7 algorithm and analysis routines (spindle_analysis.py, spindle_detection_example.py, etc.), demonstrating how outputs can further be processed and visualized.
Relationships & Data Flow
• Data Flow:
• Start from raw EEG data input, proceed through a preprocessing pipeline embedded within the scripts.
• The preprocessed data is then passed to the SUMO model (as defined in the “sumo/model” module) during training or prediction.
• Training is managed via bin/train.py where configuration files (from “config/”) determine hyperparameters and training procedures. Logging is handled by the “sumo/log” module.
• The evaluated or final trained model checkpoint is saved in “output/final.ckpt” and used by scripts like predict_plain_data.py for further prediction tasks.
• Component Interactions:
• Configuration Files (config/) feed parameters to both training & evaluation drivers (bin/ scripts) as well as the core model components (sumo/).
• The core model in “sumo/model” is supported by loss functions (sumo/loss) and evaluated by functions in “sumo/evaluation.”
• Data splits generated and managed by helper scripts or the “scripts/” directory are used to prepare training, validation, and test sets.
Architectural Patterns & Design Principles
• Modularity:
• The project is modularly divided into independent components: data handling, model definition, loss computation, evaluation, and logging.
• Configuration Over Convention:
• Centralized use of YAML configuration files provides flexibility in defining experiments and easily switching between training, evaluation, or prediction modes.
• Pipeline Structure:
• The project implements an end-to-end pipeline from data input, preprocessing, model training, evaluation, and prediction. Pytorch Lightning enforces structure and readability by encapsulating training routines.
Relevant Technologies, Frameworks & Libraries
• PyTorch Lightning: For training and evaluation routines.
• PyTorch (torch.nn, torch.optim): Underlying neural network computations, model definitions, and loss functions.
• YAML: For configuration management.
• Python standard libraries along with NumPy and helper libraries used within preprocessing and visualization scripts.
Specific Elements
• Component Labels:
• Interactions between configuration management and training/evaluation flows.
• Organizational Clarity:
• Separate the architecture into layers or zones. One zone can be external interfaces (scripts, data files), one zone for the core modules (sumo details), and one for training orchestration.
• External dependencies (e.g., PyTorch Lightning) as annotations around the training/evaluation block.