# dslaborg/sumo architecture

> Architecture diagram of the GitHub repository dslaborg/sumo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dslaborg/sumo
- Repository: https://github.com/dslaborg/sumo

## Overview

##### Main Components

- Data Input & Preprocessing:
- Input: EEG data (e.g., “input/eeg-sample.npy” and pickled subject data generated via create_data_splits.py).
- Preprocessing: Embedded in the prediction script (predict_plain_data.py) as well as within the training pipeline (downsampling, Butterworth filtering, and z-transformation).

- Configuration Management:
- YAML Files in “config/” (e.g. default.yaml, final.yaml, predict.yaml) supply experiment parameters such as model architecture, training specifics, and evaluation metrics.
- These configuration settings dynamically affect data loading, model parameters, loss functions, and training regimes.

- Training & Evaluation:
- Core driver scripts in “bin/”: train.py and eval.py which use the PyTorch Lightning framework.
- These scripts orchestrate the training process, logging (console and Tensorboard), early stopping, and performance evaluation using configurations and data splits.

- Model & Core Library:
- The “sumo/” package is the heart of the project and is divided into several modules:
- sumo/model: Contains the SUMO U-Net architecture and model partitioning.
- sumo/data: Responsible for data loaders and any data manipulation functions.
- sumo/loss: Implements the loss functions (e.g., GeneralizedDiceLoss) used during training.
- sumo/evaluation: Includes metrics and performance analysis utilities.
- sumo/log: Handles logging mechanisms across training and evaluation.
- sumo/config: Wraps configuration loading or custom parameter handling.

- Auxiliary & Analysis Scripts:
- Scripts in the “scripts/” directory include implementations for the A7 algorithm and analysis routines (spindle_analysis.py, spindle_detection_example.py, etc.), demonstrating how outputs can further be processed and visualized.

##### Relationships & Data Flow

- Data Flow:
- Start from raw EEG data input, proceed through a preprocessing pipeline embedded within the scripts.
- The preprocessed data is then passed to the SUMO model (as defined in the “sumo/model” module) during training or prediction.
- Training is managed via bin/train.py where configuration files (from “config/”) determine hyperparameters and training procedures. Logging is handled by the “sumo/log” module.
- The evaluated or final trained model checkpoint is saved in “output/final.ckpt” and used by scripts like predict_plain_data.py for further prediction tasks.

- Component Interactions:
- Configuration Files (config/) feed parameters to both training & evaluation drivers (bin/ scripts) as well as the core model components (sumo/).
- The core model in “sumo/model” is supported by loss functions (sumo/loss) and evaluated by functions in “sumo/evaluation.”
- Data splits generated and managed by helper scripts or the “scripts/” directory are used to prepare training, validation, and test sets.

##### Architectural Patterns & Design Principles

- Modularity:
- The project is modularly divided into independent components: data handling, model definition, loss computation, evaluation, and logging.

- Configuration Over Convention:
- Centralized use of YAML configuration files provides flexibility in defining experiments and easily switching between training, evaluation, or prediction modes.

- Pipeline Structure:
- The project implements an end-to-end pipeline from data input, preprocessing, model training, evaluation, and prediction. Pytorch Lightning enforces structure and readability by encapsulating training routines.

##### Relevant Technologies, Frameworks & Libraries

- PyTorch Lightning: For training and evaluation routines.
- PyTorch (torch.nn, torch.optim): Underlying neural network computations, model definitions, and loss functions.
- YAML: For configuration management.
- Python standard libraries along with NumPy and helper libraries used within preprocessing and visualization scripts.

##### Specific Elements

- Component Labels:

- Interactions between configuration management and training/evaluation flows.

- Organizational Clarity:
- Separate the architecture into layers or zones. One zone can be external interfaces (scripts, data files), one zone for the core modules (sumo details), and one for training orchestration.
- External dependencies (e.g., PyTorch Lightning) as annotations around the training/evaluation block.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Input & Preprocessing
    subgraph "Data Input & Preprocessing"
        EEGInput["EEG Data Input"]:::external
        DataSplits["Data Splits"]:::external
        Preprocessing["Preprocessing Pipeline"]:::external
    end

    %% Configuration Management
    subgraph "Configuration Management"
        YAMLDefault["default.yaml"]:::config
        YAMLFinal["final.yaml"]:::config
        YAMLPredict["predict.yaml"]:::config
        ConfigModule["Configuration Module"]:::config
    end

    %% Training & Evaluation Drivers
    subgraph "Training & Evaluation"
        TrainDriver["Training Driver"]:::external
        EvalDriver["Evaluation Driver"]:::external
        Checkpoint["Checkpoint Output (final.ckpt)"]:::external
    end

    %% Evaluation Utilities
    subgraph "Evaluation Utilities"
        PerfAnalysis["Performance Analysis"]:::internal
        EvalUtils["Evaluation Utils"]:::internal
    end

    %% Model & Core Library
    subgraph "Model & Core Library"
        SUMOModel["SUMO Model (sumo.py)"]:::internal
        ModelParts["Model Partitioning (model_parts.py)"]:::internal
        DataHandling["Data Handling"]:::internal
        LossFunctions["Loss Functions"]:::internal
        Logging["Logging"]:::internal
    end

    %% Auxiliary & Analysis Scripts
    subgraph "Auxiliary & Analysis Scripts"
        SpindleAnalysis["Spindle Analysis"]:::external
        SpindleCorrelations["Spindle Correlations"]:::external
        SpindleDetection["Spindle Detection Example"]:::external
        A7Butter["A7 Butter Filter"]:::external
        A7Features["A7 Feature Calculation"]:::external
        A7Detection["A7 Spindle Detection"]:::external
        A7Parallel["A7 Parallel Evaluation"]:::external
    end

    %% Data Flow Connections
    EEGInput -->|"feeds"| Preprocessing
    DataSplits -->|"creates splits for"| Preprocessing
    Preprocessing -->|"processed data"| SUMOModel
    Preprocessing -->|"processed data"| DataHandling

    %% Configuration Connections
    YAMLDefault -->|"configures"| TrainDriver
    YAMLFinal -->|"configures"| TrainDriver
    YAMLPredict -->|"configures"| EvalDriver
    ConfigModule -->|"supplies params to"| TrainDriver
    ConfigModule -->|"supplies params to"| EvalDriver

    %% Training Flow
    TrainDriver -->|"trains using"| SUMOModel
    TrainDriver -->|"trains using"| ModelParts
    TrainDriver -->|"employs"| LossFunctions
    TrainDriver -->|"employs"| Logging
    TrainDriver -->|"saves model to"| Checkpoint

    %% Evaluation Flow
    Checkpoint -->|"evaluated by"| EvalDriver
    EvalDriver -->|"analyzes with"| PerfAnalysis
    EvalDriver -->|"analyzes with"| EvalUtils

    %% Auxiliary Analysis Connections
    Checkpoint -->|"feeds into"| SpindleAnalysis
    Checkpoint -->|"feeds into"| SpindleCorrelations
    Checkpoint -->|"feeds into"| SpindleDetection

    EEGInput -->|"raw data processed by"| A7Butter
    A7Butter -->|"feature extraction"| A7Features
    A7Features -->|"detects spindles"| A7Detection
    A7Detection -->|"parallel evaluation"| A7Parallel

    %% Click Events for Data Input & Preprocessing
    click EEGInput "https://github.com/dslaborg/sumo/blob/main/input/eeg-sample.npy"
    click Preprocessing "https://github.com/dslaborg/sumo/blob/main/scripts/predict_plain_data.py"
    click DataSplits "https://github.com/dslaborg/sumo/blob/main/scripts/create_data_splits.py"

    %% Click Events for Configuration Management
    click YAMLDefault "https://github.com/dslaborg/sumo/blob/main/config/default.yaml"
    click YAMLFinal "https://github.com/dslaborg/sumo/blob/main/config/final.yaml"
    click YAMLPredict "https://github.com/dslaborg/sumo/blob/main/config/predict.yaml"
    click ConfigModule "https://github.com/dslaborg/sumo/blob/main/sumo/config/config.py"

    %% Click Events for Training & Evaluation
    click TrainDriver "https://github.com/dslaborg/sumo/blob/main/bin/train.py"
    click EvalDriver "https://github.com/dslaborg/sumo/blob/main/bin/eval.py"
    click PerfAnalysis "https://github.com/dslaborg/sumo/blob/main/sumo/evaluation/performance_analysis.py"
    click EvalUtils "https://github.com/dslaborg/sumo/blob/main/sumo/evaluation/utils.py"

    %% Click Events for Model & Core Library
    click SUMOModel "https://github.com/dslaborg/sumo/blob/main/sumo/model/sumo.py"
    click ModelParts "https://github.com/dslaborg/sumo/blob/main/sumo/model/model_parts.py"
    click DataHandling "https://github.com/dslaborg/sumo/blob/main/sumo/data/data.py"
    click LossFunctions "https://github.com/dslaborg/sumo/blob/main/sumo/loss/loss.py"
    click Logging "https://github.com/dslaborg/sumo/blob/main/sumo/log/log.py"

    %% Click Events for Auxiliary & Analysis Scripts
    click SpindleAnalysis "https://github.com/dslaborg/sumo/blob/main/scripts/spindle_analysis.py"
    click SpindleCorrelations "https://github.com/dslaborg/sumo/blob/main/scripts/spindle_analysis_correlations.py"
    click SpindleDetection "https://github.com/dslaborg/sumo/blob/main/scripts/spindle_detection_example.py"
    click A7Butter "https://github.com/dslaborg/sumo/blob/main/scripts/a7/butter_filter.py"
    click A7Features "https://github.com/dslaborg/sumo/blob/main/scripts/a7/calculate_features.py"
    click A7Detection "https://github.com/dslaborg/sumo/blob/main/scripts/a7/detect_spindles.py"
    click A7Parallel "https://github.com/dslaborg/sumo/blob/main/scripts/helper/a7_parallel_evaluation.py"

    %% Styles
    classDef external fill:#ffdddd,stroke:#cc0000,stroke-width:2px;
    classDef config fill:#ddffdd,stroke:#009900,stroke-width:2px;
    classDef internal fill:#ddddff,stroke:#000099,stroke-width:2px;
```
