How sanger-tol/genomenote works
This project is a Nextflow-based bioinformatics pipeline (built within the nf-core framework) that creates “genome note” documents. Its purpose is to integrate various genomic data analyses – including HiC contact map generation, metadata retrieval from public repositories (NCBI, ENA, GoaT), genome completeness evaluation using BUSCO and MerquryFK, along with additional annotation and summary statistics – and then combine all this information with a formatted template to output a complete genome note document. The project structure (with folders such as bin, modules, subworkflows, lib, and conf, alongside extensive nf-core module implementations) and the README description both indicate that the pipeline orchestrates many external tools and custom scripts as independent components.
• Input & Configuration:
• User-supplied samplesheet (CSV) and genome FASTA file.
• Configuration files (e.g., nextflow.config, conf/*) that define operational parameters.
• Workflow Orchestration:
• The main Nextflow script (main.nf) that ties everything together.
• nf-core modules and subworkflows (local modules under modules/local and modules/nf-core; subworkflows under subworkflows/local) for discrete analytical tasks.
• Data Ingestion & Metadata Retrieval:
• Custom scripts in the “bin” directory that wrap API calls to external services (NCBI, ENA, GoaT).
• API calls to public databases for assembly metadata and related data.
• Data Processing & Analysis Modules:
• HiC data processing:
• Alignment processing using samtools and bedtools (e.g., “bin/filter_bed.sh” and associated modules for converting and filtering BAM/Bed files).
• Contact map generation using Cooler (with commands such as cload, zoomify, and dump).
• Genome completeness and quality assessment:
• BUSCO, MerquryFK, and FASTK integrations.
• Annotation statistics:
• AGAT (and additional BUSCO runs) for annotation evaluations.
• Data collation:
• Python scripts, like “create_table.py” and others in bin to combine datasets and statistics.
• Reporting & Visualization:
• Generation of the final genome note document using templating resources (assets directory with template files).
• Integration with visualization tools such as MultiQC and R, plus HiGlass for displaying contact maps.
Relationships and Data Flow
• From within Nextflow, branch the execution into several parallel modules:
• One branch to “Data Ingestion & Metadata Retrieval” – highlighting custom scripts that fetch data from external APIs (annotate these with names like “fetch_ensembl_metadata.py”, “fetch_gbif_metadata.py”).
• A branch to the “HiC and Alignment Processing” module – showing components like samtools, bedtools, and the Coolers modules.
• Another branch to “Genome Completeness / Quality Metrics” – incorporating tools like BUSCO, Merqury.FK, FASTK.
• Yet another branch to “Annotation Statistics” – using AGAT and additional processing modules.
• Finally, have an output component block labeled “Reporting & Visualization” that feeds into visualization servers (e.g., HiGlass) and aggregated summary outputs (e.g., MultiQC reports).
Architectural Patterns and Design Principles
• Modular design provided by nf-core and Nextflow DSL2. Each module (or subworkflow) is self-contained, allowing independent development and testing.
• Use of external command-line tools (samtools, bedtools, Cooler, BUSCO, etc.) integrated via wrappers in the “bin” and “modules” directories.
• The design follows a pipeline pattern where discrete data transformation steps are connected via well-defined inputs/outputs, suitable for parallel processing.
• The configuration management (using Nextflow profiles, Docker/Conda/Singularity containers) is an essential part of the orchestration.