# sanger-tol/genomenote architecture

> Architecture diagram of the GitHub repository sanger-tol/genomenote, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sanger-tol/genomenote
- Repository: https://github.com/sanger-tol/genomenote

## Overview

This project is a Nextflow-based bioinformatics pipeline (built within the nf-core framework) that creates “genome note” documents. Its purpose is to integrate various genomic data analyses – including HiC contact map generation, metadata retrieval from public repositories (NCBI, ENA, GoaT), genome completeness evaluation using BUSCO and MerquryFK, along with additional annotation and summary statistics – and then combine all this information with a formatted template to output a complete genome note document. The project structure (with folders such as bin, modules, subworkflows, lib, and conf, alongside extensive nf-core module implementations) and the README description both indicate that the pipeline orchestrates many external tools and custom scripts as independent components.

- Input & Configuration:
- User-supplied samplesheet (CSV) and genome FASTA file.
- Configuration files (e.g., nextflow.config, conf/*) that define operational parameters.
- Workflow Orchestration:
- The main Nextflow script (main.nf) that ties everything together.
- nf-core modules and subworkflows (local modules under modules/local and modules/nf-core; subworkflows under subworkflows/local) for discrete analytical tasks.
- Data Ingestion & Metadata Retrieval:
- Custom scripts in the “bin” directory that wrap API calls to external services (NCBI, ENA, GoaT).
- API calls to public databases for assembly metadata and related data.
- Data Processing & Analysis Modules:
- HiC data processing:
- Alignment processing using samtools and bedtools (e.g., “bin/filter_bed.sh” and associated modules for converting and filtering BAM/Bed files).
- Contact map generation using Cooler (with commands such as cload, zoomify, and dump).
- Genome completeness and quality assessment:
- BUSCO, MerquryFK, and FASTK integrations.
- Annotation statistics:
- AGAT (and additional BUSCO runs) for annotation evaluations.
- Data collation:
- Python scripts, like “create_table.py” and others in bin to combine datasets and statistics.
- Reporting & Visualization:
- Generation of the final genome note document using templating resources (assets directory with template files).
- Integration with visualization tools such as MultiQC and R, plus HiGlass for displaying contact maps.

##### Relationships and Data Flow
- From within Nextflow, branch the execution into several parallel modules:
- One branch to “Data Ingestion & Metadata Retrieval” – highlighting custom scripts that fetch data from external APIs (annotate these with names like “fetch_ensembl_metadata.py”, “fetch_gbif_metadata.py”).
- A branch to the “HiC and Alignment Processing” module – showing components like samtools, bedtools, and the Coolers modules.
- Another branch to “Genome Completeness / Quality Metrics” – incorporating tools like BUSCO, Merqury.FK, FASTK.
- Yet another branch to “Annotation Statistics” – using AGAT and additional processing modules.
- Finally, have an output component block labeled “Reporting & Visualization” that feeds into visualization servers (e.g., HiGlass) and aggregated summary outputs (e.g., MultiQC reports).

##### Architectural Patterns and Design Principles
- Modular design provided by nf-core and Nextflow DSL2. Each module (or subworkflow) is self-contained, allowing independent development and testing.
- Use of external command-line tools (samtools, bedtools, Cooler, BUSCO, etc.) integrated via wrappers in the “bin” and “modules” directories.
- The design follows a pipeline pattern where discrete data transformation steps are connected via well-defined inputs/outputs, suitable for parallel processing.
- The configuration management (using Nextflow profiles, Docker/Conda/Singularity containers) is an essential part of the orchestration.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input & Configuration Layer
    subgraph "Input & Configuration"
        node_input1["User Samplesheet (CSV)"]
        node_input2["User Samplesheet (Full CSV)"]
        node_genome["Genome FASTA"]
        node_config["Configuration - nextflow.config"]
        node_conf["Configuration - conf Files"]
    end

    %% Workflow Orchestration Layer
    subgraph "Workflow Orchestration"
        node_orch["Nextflow Orchestrator (main.nf)"]
        subgraph "nf-core Modules"
            node_busco["BUSCO Module"]
            node_cooler["Cooler Tools"]
            node_multiqc["MultiQC Module"]
            node_fastk["FASTK Module"]
            node_agat["AGAT Module"]
        end
        subgraph "Local Modules & Subworkflows"
            node_local1["Combine Metadata Module"]
            node_filter_module["Filter Bed Module"]
            node_extract_local["Extract Annotation Stats (Local)"]
            node_createtable_module["Create Table Module"]
            node_populate_module["Populate Template Module"]
        end
    end

    %% Data Ingestion & Metadata Retrieval
    subgraph "Data Ingestion & Metadata Retrieval"
        node_fetch_ensembl["Fetch ENSEMBL Metadata (Python)"]
        node_fetch_gbif["Fetch GBIF Metadata (Python)"]
        node_parse_goat["Parse GOAT JSON Assembly"]
    end

    %% Data Processing & Analysis
    subgraph "Data Processing & Analysis"
        subgraph "HiC Data Processing"
            node_filter_bin["Filter Bed Script"]
            node_contact_map["Contact Map Generation (Cooler)"]
        end
        subgraph "Genome Completeness & Quality"
            node_busco2["BUSCO Integration"]
            node_fastk2["MerquryFK/FASTK Integration"]
        end
        subgraph "Annotation Statistics"
            node_agat2["AGAT Statistics"]
            node_extract_bin["Extract Annotation Stats (Python)"]
        end
    end

    %% Data Collation & Document Generation
    subgraph "Data Collation & Document Generation"
        node_createtable["Create Table Script"]
        node_populate["Populate Genome Note Template (Script)"]
    end

    %% Reporting & Visualization
    subgraph "Reporting & Visualization"
        node_template_docx["Genome Note Template (DOCX)"]
        node_template_xml["Genome Note Template (XML)"]
        node_higlass["HiGlass Integration"]
    end

    %% Data Flow Connections

    %% From Input & Configuration to Workflow Orchestration
    node_input1 --> node_orch
    node_config --> node_orch

    %% Branch to Data Ingestion & Metadata Retrieval
    node_orch --> node_fetch_ensembl
    node_orch --> node_fetch_gbif
    node_orch --> node_parse_goat

    %% Branch to HiC Data Processing
    node_orch --> node_filter_bin
    node_filter_bin --> node_filter_module
    node_filter_module --> node_contact_map

    %% Branch to Genome Completeness & Quality
    node_orch --> node_busco2
    node_orch --> node_fastk2

    %% Branch to Annotation Statistics
    node_orch --> node_agat2
    node_orch --> node_extract_bin
    node_orch --> node_extract_local

    %% Convergence into Data Collation & Document Generation
    node_fetch_ensembl --> node_createtable
    node_fetch_gbif --> node_createtable
    node_parse_goat --> node_createtable
    node_contact_map --> node_createtable
    node_busco2 --> node_createtable
    node_fastk2 --> node_createtable
    node_agat2 --> node_createtable
    node_extract_bin --> node_createtable
    node_extract_local --> node_createtable

    node_createtable --> node_populate
    node_populate --> node_template_docx
    node_populate --> node_template_xml
    node_populate --> node_higlass
    node_populate --> node_multiqc

    %% Integration of Local Collation Modules
    node_createtable --> node_createtable_module
    node_populate --> node_populate_module

    %% Styles
    classDef input fill:#FFCC00,stroke:#333,stroke-width:2px;
    classDef orchestrator fill:#66CCFF,stroke:#333,stroke-width:2px;
    classDef ingestion fill:#CCFF99,stroke:#333,stroke-width:2px;
    classDef hic fill:#FF9999,stroke:#333,stroke-width:2px;
    classDef quality fill:#FFB6C1,stroke:#333,stroke-width:2px;
    classDef annotation fill:#FFD700,stroke:#333,stroke-width:2px;
    classDef collation fill:#CCCCFF,stroke:#333,stroke-width:2px;
    classDef reporting fill:#99FFCC,stroke:#333,stroke-width:2px;

    class node_input1,node_input2,node_genome,node_config,node_conf input;
    class node_orch orchestrator;
    class node_fetch_ensembl,node_fetch_gbif,node_parse_goat ingestion;
    class node_filter_bin,node_filter_module,node_contact_map hic;
    class node_busco2,node_fastk2 quality;
    class node_agat2,node_extract_bin,node_extract_local annotation;
    class node_createtable,node_populate,node_createtable_module,node_populate_module collation;
    class node_template_docx,node_template_xml,node_higlass,node_multiqc reporting;

    %% Click Events
    click node_input1 "https://github.com/sanger-tol/genomenote/blob/main/assets/samplesheet.csv"
    click node_input2 "https://github.com/sanger-tol/genomenote/blob/main/assets/samplesheet_full.csv"
    click node_config "https://github.com/sanger-tol/genomenote/blob/main/nextflow.config"
    click node_conf "https://github.com/sanger-tol/genomenote/tree/main/conf/"
    click node_orch "https://github.com/sanger-tol/genomenote/blob/main/main.nf"
    click node_busco "https://github.com/sanger-tol/genomenote/tree/main/modules/nf-core/busco/"
    click node_cooler "https://github.com/sanger-tol/genomenote/tree/main/modules/nf-core/cooler/"
    click node_multiqc "https://github.com/sanger-tol/genomenote/tree/main/modules/nf-core/multiqc/"
    click node_fastk "https://github.com/sanger-tol/genomenote/tree/main/modules/nf-core/fastk/fastk"
    click node_agat "https://github.com/sanger-tol/genomenote/tree/main/modules/nf-core/agat/"
    click node_local1 "https://github.com/sanger-tol/genomenote/blob/main/modules/local/combine_metadata.nf"
    click node_fetch_ensembl "https://github.com/sanger-tol/genomenote/blob/main/bin/fetch_ensembl_metadata.py"
    click node_fetch_gbif "https://github.com/sanger-tol/genomenote/blob/main/bin/fetch_gbif_metadata.py"
    click node_parse_goat "https://github.com/sanger-tol/genomenote/blob/main/bin/parse_json_goat_assembly.py"
    click node_filter_bin "https://github.com/sanger-tol/genomenote/blob/main/bin/filter_bed.sh"
    click node_filter_module "https://github.com/sanger-tol/genomenote/blob/main/modules/local/filter/bed.nf"
    click node_busco2 "https://github.com/sanger-tol/genomenote/tree/main/modules/nf-core/busco/busco"
    click node_fastk2 "https://github.com/sanger-tol/genomenote/tree/main/modules/nf-core/fastk/fastk"
    click node_agat2 "https://github.com/sanger-tol/genomenote/tree/main/modules/nf-core/agat/spstatistics"
    click node_extract_bin "https://github.com/sanger-tol/genomenote/blob/main/bin/extract_annotation_statistics_info.py"
    click node_extract_local "https://github.com/sanger-tol/genomenote/blob/main/modules/local/extract_annotation_statistics_info.nf"
    click node_createtable "https://github.com/sanger-tol/genomenote/blob/main/bin/create_table.py"
    click node_populate "https://github.com/sanger-tol/genomenote/blob/main/bin/populate_genome_note_template.py"
    click node_createtable_module "https://github.com/sanger-tol/genomenote/blob/main/modules/local/createtable.nf"
    click node_populate_module "https://github.com/sanger-tol/genomenote/blob/main/modules/local/populate_template.nf"
    click node_template_docx "https://github.com/sanger-tol/genomenote/blob/main/assets/genome_note_template.docx"
    click node_template_xml "https://github.com/sanger-tol/genomenote/blob/main/assets/genome_note_template.xml"
    click node_higlass "https://github.com/sanger-tol/genomenote/blob/main/bin/generate_higlass_link.py"
```
