# umcugenetics/dims architecture

> Architecture diagram of the GitHub repository umcugenetics/dims, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/umcugenetics/dims
- Repository: https://github.com/umcugenetics/dims

## Overview

##### Project Type and Purpose
- Type: A bioinformatics data‐processing pipeline (Nextflow + R + HPC + containers).
- Purpose: Automate Direct Infusion Mass Spectrometry (DIMS) data processing end-to-end: raw data ingestion → preprocessing → feature extraction → reporting.

##### File Structure
- Top‐level “DIMS.nf” and “assets/workflow.nf”: the Nextflow workflow definitions.
- “CustomModules/”: repository of per‐step scripts (R, shell, etc.) invoked by Nextflow processes.
- “db/”: static reference data (HMDB .RData files, theoretical mass tables).
- “Dockerfile” + “.gitmodules”: containerization and dependency declarations.
- “run_nextflow_dims.sh”: CLI wrapper for launching the pipeline on HPC.

##### Extract README Insights
- Two “frontends” for launching: an R Shiny GUI and a CLI script.
- Execution environment: Docker (locally) or Singularity (HPC).
- External tools: ThermoRawFileParser for raw file conversion; R libraries (xcms, dplyr, etc.).
- HPC scheduler: Slurm (srun).

##### Main Components
a. User Interfaces
- R Shiny GUI (optional front‐end)
- CLI wrapper (`run_nextflow_dims.sh`)
b. Workflow Orchestrator
- Nextflow engine (`DIMS.nf` + `assets/workflow.nf`)
c. Task Modules
- CustomModules scripts (R, shell) – each Nextflow “process”
d. Data Stores
- Input storage: raw .raw files + sample sheet
- Reference DB: `db/` RData files and lookup tables
- Output storage: processed data, reports
e. Containers & Execution Environment
- Docker image (local) → Singularity image (HPC)
- External tool container: ThermoRawFileParser under `/tools`
f. Compute Infrastructure
- HPC cluster managed by Slurm
- Worker nodes / temporary storage

##### Relationships and Data Flows
- User initiates via GUI or CLI → triggers Nextflow run.
- Nextflow engine pulls container image → schedules processes on HPC nodes.
- Each process reads raw inputs + reference DB → executes CustomModules code.
- ThermoRawFileParser invoked where needed.
- Intermediate files flow through Nextflow channels → finally aggregate into output directory.
- Email notification sent on completion.

##### Architectural Patterns and Principles
- Pipeline Orchestration (Nextflow’s dataflow model)
- Containerization for reproducibility (Docker/Singularity)
- Modular design: each “process” is a self-contained script under CustomModules
- Separation of concerns: UI vs. orchestration vs. compute vs. storage

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Interfaces
    subgraph "User Interfaces"
        GUI["R Shiny GUI"]:::ui
        CLI["CLI Wrapper"]:::ui
    end

    %% Workflow Orchestrator
    subgraph "Workflow Orchestrator"
        NFmain["Nextflow Engine (DIMS.nf)"]:::orchestrator
        WorkflowDef["Workflow Definition (assets/workflow.nf)"]:::orchestrator
        Config["Workflow Config (DIMS.config)"]:::orchestrator
    end

    %% Task Modules
    subgraph "Task Modules"
        Modules["CustomModules/"]:::orchestrator
    end

    %% Data Stores
    subgraph "Data Stores"
        Raw["Raw Data (Input FS)"]:::datastore
        RefDB["Reference Data (db/)"]:::datastore
        Output["Processed Data & Reports"]:::datastore
    end

    %% Containers & Tools
    subgraph "Containers & External Tools"
        Docker["Docker Image (Dockerfile)"]:::container
        Gitmod[".gitmodules"]:::container
        ExtTool["ThermoRawFileParser Container"]:::external
    end

    %% Compute Infrastructure
    subgraph "Compute Infrastructure"
        Slurm["Slurm Scheduler"]:::compute
        Workers["Worker Nodes"]:::compute
        Temp["Temporary Storage"]:::compute
    end

    %% External Services
    Email["Email Notification Service"]:::external

    %% Connections
    GUI -->|launch pipeline| NFmain
    CLI -->|launch pipeline| NFmain

    NFmain -->|reads config| Config
    NFmain -->|includes workflow| WorkflowDef
    NFmain -->|schedules jobs on| Slurm

    Slurm --> Workers
    Workers -->|mounts for input| Raw
    Workers -->|mounts for reference| RefDB
    Workers -->|writes to| Output
    Workers -->|executes scripts from| Modules
    Modules -->|calls external tool| ExtTool

    Docker -->|provides container for| NFmain
    Docker -->|provides container for| Modules
    Docker -->|provides container for| ExtTool
    Gitmod -->|declares external tools submodule| ExtTool

    Workers -->|runs on| Docker
    NFmain -->|sends notification to| Email

    %% Click Events
    click CLI "https://github.com/umcugenetics/dims/blob/master/run_nextflow_dims.sh"
    click NFmain "https://github.com/umcugenetics/dims/blob/master/DIMS.nf"
    click WorkflowDef "https://github.com/umcugenetics/dims/blob/master/assets/workflow.nf"
    click Config "https://github.com/umcugenetics/dims/blob/master/DIMS.config"
    click Modules "https://github.com/umcugenetics/dims/tree/master/CustomModules/"
    click RefDB "https://github.com/umcugenetics/dims/tree/master/db/"
    click Docker "https://github.com/umcugenetics/dims/tree/master/Dockerfile"
    click Gitmod "https://github.com/umcugenetics/dims/blob/master/.gitmodules"
    click GUI "https://github.com/umcugenetics/dims/blob/master/README.md"
    click WorkflowDef "https://github.com/umcugenetics/dims/blob/master/assets/workflow_complete.html"

    %% Styles
    classDef ui fill:#ADD8E6,stroke:#333,stroke-width:1px;
    classDef orchestrator fill:#90EE90,stroke:#333,stroke-width:1px;
    classDef compute fill:#D3D3D3,stroke:#333,stroke-width:1px;
    classDef datastore fill:#FFDAB9,stroke:#333,stroke-width:1px;
    classDef container fill:#E6E6FA,stroke:#333,stroke-width:1px;
    classDef external fill:#FFFACD,stroke:#333,stroke-width:1px;
```
