# marbl/verkko architecture

> Architecture diagram of the GitHub repository marbl/verkko, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/marbl/verkko
- Repository: https://github.com/marbl/verkko

## Overview

##### Project Type and Purpose
- Type: Open-source, command-line bioinformatics workflow
- Purpose: Hybrid telomere-to-telomere genome assembly using high-accuracy long reads and ultra-long reads, orchestrated as a modular Snakemake pipeline

##### Main System Components
1. User Interface
– “verkko” CLI wrapper (verkko.sh)
– Command-line options (–hifi, –nano, –hic1/2, –porec, –hap-kmers, etc.)
2. Configuration
– YAML config file (verkko.yml) generated by the wrapper
– Profiles folder (src/profiles) containing slurm-sge submit/status scripts and config.yaml
3. Workflow Engine
– Snakemake core (“Snakefile” and “src/Snakefiles/*”)
– Makefiles (src/Makefile, src/Makefile.boilermake, src/main.mk)
a. Read Correction
- Uses Canu modules under src/canu & 0-correction Snakefiles
b. Multiplex de Bruijn Graph Construction
- MBG (src/MBG) invoked via Snakefile 1-buildGraph.sm
c. Graph Simplification & Processing
- Snakefiles 2-processGraph.sm, 4-processONT.sm, 5-untip.sm
d. Read Alignment
- GraphAligner for ONT (3-alignONT.sm)
e. Loop/Tangle Resolution
- Rukki module (src/rukki and 6-layoutContigs.sm)
f. Consensus Generation
- Canu consensus scripts (7-generateConsensus.sm)
g. Scaffolding & Phasing
- Hi-C/PoreC module (8-hicPipeline.sm)
- Rukki for trio phasing (scripts under src/scripts and src/scripts/scaffolding)
5. External Tools & Libraries
– Canu, GraphAligner, MashMap, Winnowmap, Minimap2, BWA, Samtools, seqtk, meryl, networkx, Python, Rust, GCC
6. Execution Environment
– Local machine or Grid (Slurm, SGE, PBS) via profile scripts
– Conda environment or direct binaries under verkko/lib/verkko/bin
7. Data Stores & Outputs
– Input read files (HiFi, ONT, Hi-C/PoreC, hap-mer DBs)
– Intermediate GFA/CSV files under asm/*
– Final assembly: asm/assembly.fasta, .gfa, .scfmap, .haplotype*.fasta

##### Relationships and Data Flow
- CLI wrapper → generates config → invokes Snakemake
- Snakemake → orchestrates directed-acyclic-graph of Snakefiles
- Profiles scripts → translate Snakemake jobs to grid scheduler submissions

##### Architectural Patterns & Design Principles
- Workflow-as-Code: Snakemake Snakefiles define tasks and dependencies
- Modular Decomposition: each pipeline stage is encapsulated in its own Snakefile + scripts
- Extensibility: users can drop new scripts into src/scripts or add custom Snakefiles
- Scalability: grid integration via profiles, resource parameters per stage

##### Essential Elements
– Notes on where external tools are invoked
– Visualization of profiles hooking into grid scheduler

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Layer
    subgraph "User Layer"
        CLI["verkko.sh (CLI Wrapper)"]:::core
    end

    %% Configuration Layer
    subgraph "Configuration Layer"
        CFG_GEN["Generate verkko.yml"]:::core
        PROFILES["Profiles Scripts"]:::core
    end

    %% Workflow Engine
    subgraph "Workflow Layer"
        SNAKE_ENGINE["Snakemake Engine<br/>(Snakefile + Makefiles)"]:::core
    end

    %% Pipeline Stages
    subgraph "Pipeline Stages"
        RC["1. Read Correction"]:::stage
        DBG["2. de Bruijn Graph Construction"]:::stage
        GS["3-5. Graph Simplification & Processing"]:::stage
        RA["3. Read Alignment (ONT)"]:::stage
        TR["6. Loop/Tangle Resolution (Rukki)"]:::stage
        CG["7. Consensus Generation"]:::stage
        SP["8. Scaffolding & Phasing"]:::stage
    end

    %% External Tools
    subgraph "External Tools" 
        ET["Canu, GraphAligner, MashMap,..."]:::tool
    end

    %% Execution Layer
    subgraph "Execution Layer"
        EXEC["Grid Scheduler via Profiles"]:::core
    end

    %% Data Layer
    subgraph "Data Layer"
        INPUTS["Raw Reads (HiFi, ONT, Hi-C/PoreC)"]:::storage
        INTER["Intermediate GFA/CSV<br/>asm/*"]:::storage
        OUTPUT["Final Assembly<br/>asm/assembly.fasta,..."]:::storage
    end

    %% Connections
    CLI -->|invoke| CFG_GEN
    CLI -->|invoke| SNAKE_ENGINE
    CFG_GEN -->|writes config| SNAKE_ENGINE
    PROFILES -->|used by| SNAKE_ENGINE
    SNAKE_ENGINE -->|runs| RC
    SNAKE_ENGINE -->|runs| DBG
    SNAKE_ENGINE -->|runs| GS
    SNAKE_ENGINE -->|runs| RA
    SNAKE_ENGINE -->|runs| TR
    SNAKE_ENGINE -->|runs| CG
    SNAKE_ENGINE -->|runs| SP

    INPUTS -->|reads| RC
    RC -->|corrected reads| DBG
    DBG -->|unitig.gfa| GS
    GS -->|processed graph| RA
    RA -->|alignments| TR
    TR -->|layout| CG
    CG -->|consensus| SP
    SP -->|scaffolds| OUTPUT

    EXEC -->|submits jobs| SNAKE_ENGINE

    %% Click Events
    click CLI "https://github.com/marbl/verkko/blob/master/src/verkko.sh"
    click PROFILES "https://github.com/marbl/verkko/blob/master/src/profiles/config.yaml"
    click PROFILES "https://github.com/marbl/verkko/blob/master/src/profiles/jobscript.sh"
    click PROFILES "https://github.com/marbl/verkko/blob/master/src/profiles/slurm-sge-status.sh"
    click PROFILES "https://github.com/marbl/verkko/blob/master/src/profiles/slurm-sge-submit.sh"
    click SNAKE_ENGINE "https://github.com/marbl/verkko/tree/master/src/Snakefile"
    click SNAKE_ENGINE "https://github.com/marbl/verkko/tree/master/src/Snakefiles/"
    click SNAKE_ENGINE "https://github.com/marbl/verkko/tree/master/src/Makefile"
    click SNAKE_ENGINE "https://github.com/marbl/verkko/blob/master/src/Makefile.boilermake"
    click SNAKE_ENGINE "https://github.com/marbl/verkko/blob/master/src/main.mk"
    click RC "https://github.com/marbl/verkko/tree/master/src/canu/"
    click DBG "https://github.com/marbl/verkko/tree/master/src/MBG/"
    click DBG "https://github.com/marbl/verkko/blob/master/src/Snakefiles/1-buildGraph.sm"
    click GS "https://github.com/marbl/verkko/blob/master/src/Snakefiles/2-processGraph.sm"
    click GS "https://github.com/marbl/verkko/blob/master/src/Snakefiles/4-processONT.sm"
    click GS "https://github.com/marbl/verkko/blob/master/src/Snakefiles/5-untip.sm"
    click RA "https://github.com/marbl/verkko/blob/master/src/Snakefiles/3-alignONT.sm"
    click TR "https://github.com/marbl/verkko/tree/master/src/rukki/"
    click TR "https://github.com/marbl/verkko/blob/master/src/Snakefiles/6-layoutContigs.sm"
    click TR "https://github.com/marbl/verkko/blob/master/src/Snakefiles/6-rukki.sm"
    click CG "https://github.com/marbl/verkko/blob/master/src/Snakefiles/7-generateConsensus.sm"
    click CG "https://github.com/marbl/verkko/blob/master/src/Snakefiles/7-combineConsensus.sm"
    click SP "https://github.com/marbl/verkko/blob/master/src/Snakefiles/8-hicPipeline.sm"
    click SP "https://github.com/marbl/verkko/blob/master/src/scripts/launch_scaffolding.py"
    click SP "https://github.com/marbl/verkko/blob/master/src/scripts/launch_phasing.py"
    click SP "https://github.com/marbl/verkko/tree/master/src/scripts/scaffolding/"

    %% Styles
    classDef core fill:#D0E8FF,stroke:#2196F3,stroke-width:2px;
    classDef stage fill:#FFE0B2,stroke:#FB8C00,stroke-width:2px;
    classDef tool fill:#C8E6C9,stroke:#388E3C,stroke-width:2px;
    classDef storage fill:#E0E0E0,stroke:#9E9E9E,stroke-width:2px;
```
