# aagi-aus/pacb-flow architecture

> Architecture diagram of the GitHub repository aagi-aus/pacb-flow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aagi-aus/pacb-flow
- Repository: https://github.com/aagi-aus/pacb-flow

## Overview

##### Main Components

a) Input Layer:
- Long reads data source
- Short reads data source
- Reference genome (optional)
- Manifest file (samples.csv)

b) Pipeline Stages:
- Primary Assembly Stage
- Scaffolding Stage (optional)
- Polishing Stage
- Coverage Analysis Stage

c) Infrastructure Components:
- Nextflow Runtime Environment
- HPC/Supercomputer Environment
- Container System (Singularity)

**┌─────────────────────────────────────────────────────────────┐**
- Input Layer                           │
- ┘
- ↓
**┌─────────────────────────────────────────────────────────────┐**
- Workflow Controller                       │
- (Nextflow main.nf runner)                    │
- ┘
- ↓
**┌─────────────────────────────────────────────────────────────┐**
- Processing Stages                         │
- ┘
- ↓
**┌─────────────────────────────────────────────────────────────┐**
- Output Layer                            │
- ┘

##### Detailed Component Breakdown

The Processing Stages should be expanded to show:

a) Primary Assembly (CANU):
- Read trimming
- Read correction
- Assembly

b) Scaffolding (Optional):
- ntLink (long read scaffolding)
- ntJoin (reference-based scaffolding)

c) Polishing:
- Polypolish module

d) Coverage Analysis:
- BWA mapping
- Bedtools coverage
- UCSC utilities

##### Key Relationships to Highlight

- Data flow between stages
- Optional pathway activation based on parameters
- Resource allocation points
- Parallel execution possibilities

##### Technical Details

- Module dependencies
- Configuration profiles (pawsey_setonix, singularity)
- Resource requirements (>100GB memory peak)
- Input/Output file formats

##### Infrastructure Layer

Relationship with:
- HPC environment
- Container runtime
- Storage systems
- Resource managers

##### Metadata

A metadata section with:
- Pipeline version
- Required Nextflow version (>22.10.7)
- Author information
- License information

Remember to include subworkflow relationships as shown in the file structure, particularly the relationship between the main workflow and local subworkflows in the 'subworkflows/local' directory.

This system design should provide a clear understanding of both the high-level architecture and the detailed workflow steps, making it valuable for both new team members and experienced developers maintaining the pipeline.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Input Layer
    subgraph Input
        LR[Long Reads]:::input
        SR[Short Reads]:::input
        RG[Reference Genome]:::input
        MF["Manifest File"]:::input
    end
    click MF "https://github.com/AAGI-AUS/PacB-flow/blob/main/testrun_manifest/samples.csv"

    %% Workflow Controller
    subgraph Controller
        NF["Nextflow Runner"]:::controller
        MPW["Main Pipeline Workflow"]:::controller
        ASW["Assembly Subworkflow"]:::controller
    end
    click NF "https://github.com/AAGI-AUS/PacB-flow/blob/main/main.nf"
    click MPW "https://github.com/AAGI-AUS/PacB-flow/blob/main/workflows/run_pacbassembly.nf"
    click ASW "https://github.com/AAGI-AUS/PacB-flow/blob/main/subworkflows/local/run_pacbassembly/main.nf"

    %% Processing Stages
    subgraph Processing
        direction TB
        
        subgraph "Primary Assembly"
            CANU["CANU Assembly"]:::process
        end
        click CANU "https://github.com/AAGI-AUS/PacB-flow/blob/main/modules/local/software/canu/assemblereads.nf"

        subgraph "Scaffolding"
            NTL["ntLink"]:::optional
            NTJ["ntJoin"]:::optional
        end
        click NTL "https://github.com/AAGI-AUS/PacB-flow/blob/main/modules/local/software/ntlink/scaffoldassembly.nf"
        click NTJ "https://github.com/AAGI-AUS/PacB-flow/blob/main/modules/local/software/ntjoin/scaffoldassembly.nf"

        subgraph "Polishing"
            PP["Polypolish"]:::process
            PPW["Polishing Workflow"]:::process
        end
        click PP "https://github.com/AAGI-AUS/PacB-flow/blob/main/modules/local/software/polypolish/polishgenome.nf"
        click PPW "https://github.com/AAGI-AUS/PacB-flow/blob/main/subworkflows/local/run_polypolish/main.nf"

        subgraph "Coverage Analysis"
            BWA["BWA Mapping"]:::process
            BT["Bedtools Coverage"]:::process
            UCSC["UCSC Utilities"]:::process
            CAW["Coverage Analysis Workflow"]:::process
        end
        click BWA "https://github.com/AAGI-AUS/PacB-flow/blob/main/modules/local/software/bwa/mapallreads.nf"
        click BT "https://github.com/AAGI-AUS/PacB-flow/blob/main/modules/local/software/bedtools/genomecoverage.nf"
        click UCSC "https://github.com/AAGI-AUS/PacB-flow/blob/main/modules/local/software/ucsc-utilities/bedgraphtobigwig.nf"
        click CAW "https://github.com/AAGI-AUS/PacB-flow/blob/main/subworkflows/local/calculate_coverage/main.nf"
    end

    %% Infrastructure Layer
    subgraph Infrastructure
        HPC["HPC Environment"]:::infra
        SING["Singularity Runtime"]:::infra
        CONF["Configuration Profiles"]:::infra
    end
    click CONF "https://github.com/AAGI-AUS/PacB-flow/blob/main/conf/profiles/pawsey_setonix.conf"

    %% Final Processing
    FP["Cleanup & Final Processing"]:::output
    click FP "https://github.com/AAGI-AUS/PacB-flow/blob/main/subworkflows/local/cleanup_genomes_final/main.nf"

    %% Connections
    Input --> Controller
    Controller --> Processing
    Processing --> FP
    Infrastructure -.-> Controller
    Infrastructure -.-> Processing

    %% Styles
    classDef input fill:#a8d1ff,stroke:#333,stroke-width:2px
    classDef controller fill:#ffe066,stroke:#333,stroke-width:2px
    classDef process fill:#90EE90,stroke:#333,stroke-width:2px
    classDef optional fill:#90EE90,stroke:#333,stroke-width:2px,stroke-dasharray: 5 5
    classDef infra fill:#d3d3d3,stroke:#333,stroke-width:2px
    classDef output fill:#ff9999,stroke:#333,stroke-width:2px

    %% Legend
    subgraph Legend
        L1["Input Component"]:::input
        L2["Controller Component"]:::controller
        L3["Process Component"]:::process
        L4["Optional Component"]:::optional
        L5["Infrastructure Component"]:::infra
        L6["Output Component"]:::output
    end
```
