How barrydigby/circrna works
Main Components
• Nextflow Engine & DSL2 Workflow: At the core is the Nextflow execution engine (using DSL2) orchestrating the pipeline. The main workflow files (main.nf in the root; workflows/circrna.nf and sub-workflows) describe the pipeline’s structure.
• Pipeline Modules: The workflow is organized into modular steps. These include:
– Pre-processing (adapter trimming with BBDUK, QC using FastQC and MultiQC)
– Reference genome and database downloads (Gencode, miRbase, TargetScan)
– CircRNA quantification using multiple tools (STAR > CIRCexplorer2, circRNA finder; HISAT2 > CIRI2 then BWA > CIRIquant; Bowtie-based methods; MapSplice, etc.)
– miRNA target prediction (using miRanda and TargetScan)
– Differential expression analysis (using DESeq2 in R with supporting scripts in the bin folder)
• Containerization & Environment: The pipeline uses Docker/Singularity images as evidenced by the Dockerfile, containers directory (with its own Dockerfile and environment.yml), and references in the README.
• Configurations & Modules: Config files in the conf directory provide parameters for various computational settings. The “lib” and “modules” directories include shared functions and nf-core modules that guide the workflow’s behavior.
• CI/CD & Documentation: The .github directory with GitHub action workflows, linting, and deployment scripts shows the continuous integration and testing architecture. Documentation under docs and assets (including images for the workflow and email templates) supports user and developer understanding.
Relationships and Data Flow
• Data Input: Start with raw RNA-Seq fastq files (and samplesheets provided in assets) that feed into the pipeline.
• Pre-processing: Data enters initial QC, adapter trimming, and filtering components.
• Parallel Processing: The pipeline splits into several circRNA detection strategies (using different aligners and algorithms) running in parallel – each encapsulated as its own module.
• Aggregation & Post-processing: The results from multiple circRNA tools are aggregated and further processed, culminating in miRNA target prediction and differential expression analysis.
• Output: Final outputs (quantification matrices, DESeq2 analysis reports, QC reports) are stored and made available for review.
Architectural Patterns & Design Principles
• Modular Architecture: The pipeline is organized into discrete modules (with several directories dedicated to modules/nf-core and local modules) that can be developed, maintained, or replaced independently.
• Containerized Process Isolation: Each process uses its own container (as promoted by Nextflow DSL2) that encapsulates dependencies, improving reproducibility and portability.
• Configuration Driven: The use of multiple configuration files (in conf and nextflow.config) enables environment-specific setups and scalability across infrastructures.
• CI/CD Integration: Automated testing (referenced by AWS CI workflows and branch workflows) ensures that each change is continuously validated.
Relevant Technologies and Frameworks
• Nextflow (DSL2) as the workflow engine.
• Docker, Singularity, and other container platforms for process isolation.
• R and its libraries (DESeq2, ggplot2, circlize, etc.) for downstream analysis.
• Multiple bioinformatics tools and aligners (STAR, HISAT2, Bowtie, etc.) integrated as separate modules.
• GitHub Actions for continuous integration/testing.
• Configuration management files (.editorconfig, nextflow.config) and version control via Git for pipeline reproducibility.
Summing Up
• The top layer representing the Nextflow Controller (orchestrator) along with input triggers.
• The bottom layer containing output data, log files, and reporting (documentation, and CI reports).
• Side elements should include the project’s configuration and container management systems.