How marbl/verkko works
Project Type and Purpose
• Type: Open-source, command-line bioinformatics workflow
• Purpose: Hybrid telomere-to-telomere genome assembly using high-accuracy long reads and ultra-long reads, orchestrated as a modular Snakemake pipeline
Main System Components
1. User Interface
– “verkko” CLI wrapper (verkko.sh)
– Command-line options (–hifi, –nano, –hic1/2, –porec, –hap-kmers, etc.)
2. Configuration
– YAML config file (verkko.yml) generated by the wrapper
– Profiles folder (src/profiles) containing slurm-sge submit/status scripts and config.yaml
3. Workflow Engine
– Snakemake core (“Snakefile” and “src/Snakefiles/*”)
– Makefiles (src/Makefile, src/Makefile.boilermake, src/main.mk)
a. Read Correction
• Uses Canu modules under src/canu & 0-correction Snakefiles
b. Multiplex de Bruijn Graph Construction
• MBG (src/MBG) invoked via Snakefile 1-buildGraph.sm
c. Graph Simplification & Processing
• Snakefiles 2-processGraph.sm, 4-processONT.sm, 5-untip.sm
d. Read Alignment
• GraphAligner for ONT (3-alignONT.sm)
e. Loop/Tangle Resolution
• Rukki module (src/rukki and 6-layoutContigs.sm)
f. Consensus Generation
• Canu consensus scripts (7-generateConsensus.sm)
g. Scaffolding & Phasing
• Hi-C/PoreC module (8-hicPipeline.sm)
• Rukki for trio phasing (scripts under src/scripts and src/scripts/scaffolding)
5. External Tools & Libraries
– Canu, GraphAligner, MashMap, Winnowmap, Minimap2, BWA, Samtools, seqtk, meryl, networkx, Python, Rust, GCC
6. Execution Environment
– Local machine or Grid (Slurm, SGE, PBS) via profile scripts
– Conda environment or direct binaries under verkko/lib/verkko/bin
7. Data Stores & Outputs
– Input read files (HiFi, ONT, Hi-C/PoreC, hap-mer DBs)
– Intermediate GFA/CSV files under asm/*
– Final assembly: asm/assembly.fasta, .gfa, .scfmap, .haplotype*.fasta
Relationships and Data Flow
• CLI wrapper → generates config → invokes Snakemake
• Snakemake → orchestrates directed-acyclic-graph of Snakefiles
• Profiles scripts → translate Snakemake jobs to grid scheduler submissions
Architectural Patterns & Design Principles
• Workflow-as-Code: Snakemake Snakefiles define tasks and dependencies
• Modular Decomposition: each pipeline stage is encapsulated in its own Snakefile + scripts
• Extensibility: users can drop new scripts into src/scripts or add custom Snakefiles
• Scalability: grid integration via profiles, resource parameters per stage
Essential Elements
– Notes on where external tools are invoked
– Visualization of profiles hooking into grid scheduler