# barrydigby/circrna architecture

> Architecture diagram of the GitHub repository barrydigby/circrna, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/barrydigby/circrna
- Repository: https://github.com/barrydigby/circrna

## Overview

##### Main Components
- Nextflow Engine & DSL2 Workflow: At the core is the Nextflow execution engine (using DSL2) orchestrating the pipeline. The main workflow files (main.nf in the root; workflows/circrna.nf and sub-workflows) describe the pipeline’s structure.
- Pipeline Modules: The workflow is organized into modular steps. These include:
– Pre-processing (adapter trimming with BBDUK, QC using FastQC and MultiQC)
– Reference genome and database downloads (Gencode, miRbase, TargetScan)
– CircRNA quantification using multiple tools (STAR > CIRCexplorer2, circRNA finder; HISAT2 > CIRI2 then BWA > CIRIquant; Bowtie-based methods; MapSplice, etc.)
– miRNA target prediction (using miRanda and TargetScan)
– Differential expression analysis (using DESeq2 in R with supporting scripts in the bin folder)
- Containerization & Environment: The pipeline uses Docker/Singularity images as evidenced by the Dockerfile, containers directory (with its own Dockerfile and environment.yml), and references in the README.
- Configurations & Modules: Config files in the conf directory provide parameters for various computational settings. The “lib” and “modules” directories include shared functions and nf-core modules that guide the workflow’s behavior.
- CI/CD & Documentation: The .github directory with GitHub action workflows, linting, and deployment scripts shows the continuous integration and testing architecture. Documentation under docs and assets (including images for the workflow and email templates) supports user and developer understanding.

##### Relationships and Data Flow
- Data Input: Start with raw RNA-Seq fastq files (and samplesheets provided in assets) that feed into the pipeline.
- Pre-processing: Data enters initial QC, adapter trimming, and filtering components.
- Parallel Processing: The pipeline splits into several circRNA detection strategies (using different aligners and algorithms) running in parallel – each encapsulated as its own module.
- Aggregation & Post-processing: The results from multiple circRNA tools are aggregated and further processed, culminating in miRNA target prediction and differential expression analysis.
- Output: Final outputs (quantification matrices, DESeq2 analysis reports, QC reports) are stored and made available for review.

##### Architectural Patterns & Design Principles
- Modular Architecture: The pipeline is organized into discrete modules (with several directories dedicated to modules/nf-core and local modules) that can be developed, maintained, or replaced independently.
- Containerized Process Isolation: Each process uses its own container (as promoted by Nextflow DSL2) that encapsulates dependencies, improving reproducibility and portability.
- Configuration Driven: The use of multiple configuration files (in conf and nextflow.config) enables environment-specific setups and scalability across infrastructures.
- CI/CD Integration: Automated testing (referenced by AWS CI workflows and branch workflows) ensures that each change is continuously validated.

##### Relevant Technologies and Frameworks
- Nextflow (DSL2) as the workflow engine.
- Docker, Singularity, and other container platforms for process isolation.
- R and its libraries (DESeq2, ggplot2, circlize, etc.) for downstream analysis.
- Multiple bioinformatics tools and aligners (STAR, HISAT2, Bowtie, etc.) integrated as separate modules.
- GitHub Actions for continuous integration/testing.
- Configuration management files (.editorconfig, nextflow.config) and version control via Git for pipeline reproducibility.

##### Summing Up
- The top layer representing the Nextflow Controller (orchestrator) along with input triggers.
- The bottom layer containing output data, log files, and reporting (documentation, and CI reports).
- Side elements should include the project’s configuration and container management systems.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Workflow Orchestration Layer
    subgraph "Workflow Orchestration"
        NFMain["Nextflow Engine: main.nf"]:::orchestration
        CircrnaWorkflow["Workflow: circrna.nf"]:::orchestration
    end
    NFMain --> CircrnaWorkflow
    click NFMain "https://github.com/barrydigby/circrna/blob/dev/main.nf"
    click CircrnaWorkflow "https://github.com/barrydigby/circrna/blob/dev/workflows/circrna.nf"

    %% Input Layer
    subgraph "Input Data"
        RawData["Raw RNA-Seq Fastq & Samplesheets"]:::data
    end
    CircrnaWorkflow --> RawData

    %% Pre-processing Layer
    subgraph "Pre-processing"
        AdapterTrimming["Adapter Trimming (BBDUK, adapters)"]:::preprocess
        QCFastQC["QC: FastQC"]:::preprocess
        QCMultiQC["QC: MultiQC"]:::preprocess
        SampleSheetCheck["Samplesheet Validation"]:::preprocess
    end
    RawData --> SampleSheetCheck
    RawData --> AdapterTrimming
    AdapterTrimming --> QCFastQC
    AdapterTrimming --> QCMultiQC
    click QCFastQC "https://github.com/barrydigby/circrna/tree/dev/modules/nf-core/modules/fastqc"
    click QCMultiQC "https://github.com/barrydigby/circrna/tree/dev/modules/nf-core/modules/multiqc"
    click AdapterTrimming "https://github.com/barrydigby/circrna/blob/dev/bin/adapters.fa"
    click SampleSheetCheck "https://github.com/barrydigby/circrna/blob/dev/modules/local/samplesheet_check.nf"

    %% CircRNA Quantification Layer
    subgraph "CircRNA Quantification"
        circFind["circRNA Detection (find_circ.py, find_circ.sh)"]:::quant
        CIRCexplorer2["CIRCexplorer2 Filtering"]:::quant
        CIRIquant["CIRIquant Filtering"]:::quant
        CircMatrix["Aggregation: circRNA Counts Matrix"]:::quant
        AlgoUnion["Aggregation: Algorithms Union"]:::quant
        AlgoIntersect["Aggregation: Algorithms Intersection"]:::quant
    end
    QCMultiQC --> circFind
    SampleSheetCheck --> circFind
    circFind --> CircMatrix
    circFind --> CIRCexplorer2
    circFind --> CIRIquant
    CIRCexplorer2 --> AlgoUnion
    CIRIquant --> AlgoIntersect
    click circFind "https://github.com/barrydigby/circrna/blob/dev/bin/find_circ.py"
    click CIRCexplorer2 "https://github.com/barrydigby/circrna/blob/dev/bin/filter_circexplorer2.sh"
    click CIRIquant "https://github.com/barrydigby/circrna/blob/dev/bin/filter_CIRIquant.sh"
    click CircMatrix "https://github.com/barrydigby/circrna/blob/dev/bin/circRNA_counts_matrix.py"
    click AlgoUnion "https://github.com/barrydigby/circrna/blob/dev/bin/consolidate_algorithms_union.R"
    click AlgoIntersect "https://github.com/barrydigby/circrna/blob/dev/bin/consolidate_algorithms_intersection.R"

    %% miRNA Target Prediction Layer
    subgraph "miRNA Target Prediction"
        TargetCore["TargetScan Core (targetscan_70.pl)"]:::miRNA
        TargetExtensions["TargetScan Extensions (associated scripts)"]:::miRNA
        MatureSeq["Mature miRNA Sequence Extraction"]:::miRNA
    end
    AlgoUnion --> TargetCore
    AlgoIntersect --> TargetExtensions
    TargetCore --> MatureSeq
    click TargetCore "https://github.com/barrydigby/circrna/blob/dev/bin/targetscan_70.pl"
    click TargetExtensions "https://github.com/barrydigby/circrna/blob/dev/bin/targetscan_70_BL_PCT.pl"
    click MatureSeq "https://github.com/barrydigby/circrna/blob/dev/bin/get_mature_seq.sh"

    %% Differential Expression Analysis Layer
    subgraph "Differential Expression Analysis"
        DEAnalysis["DE Analysis (DE.R)"]:::de
        DEAAnalysis["DEA Analysis (DEA.R)"]:::de
        Annotation["Annotation (annotate_de.nf)"]:::de
        Reporting["Reporting (annotate_report.R)"]:::de
        CountReformat["Count Matrix Reformatting"]:::de
    end
    MatureSeq --> DEAnalysis
    MatureSeq --> DEAAnalysis
    DEAnalysis --> Annotation
    DEAAnalysis --> Reporting
    Annotation --> CountReformat
    click DEAnalysis "https://github.com/barrydigby/circrna/blob/dev/bin/DE.R"
    click DEAAnalysis "https://github.com/barrydigby/circrna/blob/dev/bin/DEA.R"
    click Annotation "https://github.com/barrydigby/circrna/blob/dev/annotate_de.nf"
    click Reporting "https://github.com/barrydigby/circrna/blob/dev/annotate_report.R"
    click CountReformat "https://github.com/barrydigby/circrna/blob/dev/reformat_count_matrix.R"

    %% Output Layer
    subgraph "Output"
        FinalOutput["Final Outputs & Reports"]:::output
    end
    CountReformat --> FinalOutput
    Reporting --> FinalOutput

    %% Supporting Infrastructure
    subgraph "Supporting Infrastructure"
        %% Containerization & Environment
        subgraph "Containerization & Environment"
            Dockerfile["Dockerfile"]:::container
            ContainersPy["containers/py3/"]:::container
            EnvYML["environment.yml"]:::container
        end
        %% Configurations & Modules
        subgraph "Configurations & Modules"
            Conf["conf/"]:::config
            NextflowConfig["nextflow.config"]:::config
            Lib["lib/"]:::config
            ModulesShared["modules/"]:::config
        end
        %% CI/CD & Documentation
        subgraph "CI/CD & Documentation"
            Github[".github/"]:::ci
            Docs["docs/"]:::ci
            Assets["assets/"]:::ci
        end
    end
    NFMain --- NextflowConfig
    CircrnaWorkflow --- Lib
    FinalOutput --- Conf

    click Dockerfile "https://github.com/barrydigby/circrna/tree/dev/Dockerfile"
    click ContainersPy "https://github.com/barrydigby/circrna/tree/dev/containers/py3/"
    click EnvYML "https://github.com/barrydigby/circrna/blob/dev/environment.yml"
    click Conf "https://github.com/barrydigby/circrna/tree/dev/conf/"
    click NextflowConfig "https://github.com/barrydigby/circrna/blob/dev/nextflow.config"
    click Lib "https://github.com/barrydigby/circrna/tree/dev/lib/"
    click ModulesShared "https://github.com/barrydigby/circrna/tree/dev/modules/"
    click Github "https://github.com/barrydigby/circrna/tree/dev/.github/"
    click Docs "https://github.com/barrydigby/circrna/tree/dev/docs/"
    click Assets "https://github.com/barrydigby/circrna/tree/dev/assets/"

    %% Class definitions for coloring
    classDef orchestration fill:#add8e6,stroke:#000,stroke-width:2px;
    classDef data fill:#ffd700,stroke:#000,stroke-width:2px;
    classDef preprocess fill:#90ee90,stroke:#000,stroke-width:2px;
    classDef quant fill:#ffdab9,stroke:#000,stroke-width:2px;
    classDef miRNA fill:#ffb6c1,stroke:#000,stroke-width:2px;
    classDef de fill:#d3d3d3,stroke:#000,stroke-width:2px;
    classDef output fill:#dda0dd,stroke:#000,stroke-width:2px;
    classDef container fill:#e6e6fa,stroke:#000,stroke-width:2px;
    classDef config fill:#fafad2,stroke:#000,stroke-width:2px;
    classDef ci fill:#f0e68c,stroke:#000,stroke-width:2px;
```
