# mahdi-dcs/ms-genomics-blog architecture

> Architecture diagram of the GitHub repository mahdi-dcs/ms-genomics-blog, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mahdi-dcs/ms-genomics-blog
- Repository: https://github.com/mahdi-dcs/ms-genomics-blog

## Overview

##### Main Components
- Infrastructure Deployment:
- Bicep Templates (located under the “infra” folder) for provisioning resources such as Batch Pools, Storage Accounts, and associated modules (batch, key-vault, roles, storage).
- Assumed external components (Virtual Network, Private Endpoints Subnet, DNS Zones) that are part of the Enterprise Landing Zone.
- Storage Accounts:
- One storage account for pipeline assets and source code (with versioning enabled, HNS disabled).
- One ADLS Gen2 account for Nextflow I/O files, supporting file-level access control.
- Compute Resources:
- Two Azure Batch Pools:
- “Nextflow Runners” (small VMs to run Nextflow commands)
- “Pipeline Runners” (larger VMs for individual pipeline tasks)
- Containerization:
- A Dockerfile builds a custom Nextflow image (using the official Nextflow image plus required Python libraries), ultimately pushed to an Azure Container Registry (ACR).
- Nextflow Pipeline:
- Main script (nextflow/main.nf) and a set of modules (in the “nextflow/modules” directory) for various tasks (alignment, variant calling, etc.).
- Subworkflows (in “nextflow/subworkflows”) to handle particular pipelines for CMV and HBV.
- Data and References:
- Sequencing samples are stored in the “samples” directory.
- Reference data (including FASTA files, blast databases, and other genomic resources) reside in the “nextflow-references” folder.
- Job Submission:
- A Python submission script (submit_job.py) to upload pipeline codes and submit a Batch job.
- CI/CD Pipeline:
- A GitHub Actions workflow (in .github/workflows/deploy-infrastructure.yaml) that handles deployment of the infrastructure using the Bicep templates.

##### Relationships and Interactions
- The Bicep templates deploy the core infrastructure, including storage accounts, Batch pools, and key vaults. These resources form the backbone of the deployment environment.
- The GitHub Actions workflow automates the deployment by triggering on code commits and executing the Bicep templates.
- The Dockerfile is used to build a custom container image that is then pushed to the ACR; this image is later utilized by the Nextflow runner nodes on the Batch pools.
- The Nextflow pipeline (with its main script and modular sub-scripts) is executed on the Batch pools. It interacts with:
- The I/O storage account to read sample data and reference files (uploaded from “samples” and “nextflow-references”).
- The assets storage account which holds pipeline code and possibly configuration for reproducibility.
- The submit_job.py script ties it together by orchestrating job submission; it ensures that the appropriate environment variables (pointing to storage accounts, Batch account, key vault, etc.) are set before triggering the pipeline.

##### Architectural Patterns and Design Principles
- Infrastructure as Code: The complete infrastructure is defined through Bicep templates (in the “infra” folder), ensuring repeatable and version-controlled deployments.
- Containerization: The use of a custom Docker image allows consistency across execution environments on Azure Batch.
- Separation of Concerns: Distinct components and data flows separate the pipeline execution (Nextflow) from infrastructure provisioning (Bicep) and CI/CD processes (GitHub Actions).
- Secure Access and Scalability: Managed identities, controlled access via Key Vault, and leveraging Azure Batch pools reflect an emphasis on secure, scalable, and manageable deployment.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CI/CD Layer
    A["GitHub Actions CI/CD"]:::cicd
    click A "https://github.com/mahdi-dcs/ms-genomics-blog/blob/main/.github/workflows/deploy-infrastructure.yaml"

    %% Infrastructure Layer
    B["Bicep Infrastructure"]:::bicep
    click B "https://github.com/mahdi-dcs/ms-genomics-blog/blob/main/infra/main.bicep"
    
    A -->|"DeploysInfraViaBicep"| B

    subgraph "Infrastructure Resources"
        BA["Storage Account: Pipeline Assets<br/>(versioned)"]:::storage
        BB["Storage Account: I/O (ADLS Gen2)"]:::storage
        BC["Azure Batch Pools"]:::batch
        BD["Key Vault"]:::kv
        BE["ACR"]:::acr
    end

    B -->|"Deploys"| BA
    B -->|"Deploys"| BB
    B -->|"Deploys"| BC
    B -->|"Deploys"| BD
    B -->|"Deploys"| BE

    %% Azure Batch Pools Subdivision
    subgraph "Azure Batch Pools"
        C1["Nextflow Runners (small VMs)"]:::batch
        C2["Pipeline Runners (large VMs)"]:::batch
    end
    BC --> C1
    BC --> C2
    click BC "https://github.com/mahdi-dcs/ms-genomics-blog/tree/main/infra/modules/batch"

    %% Containerization and Job Submission
    D["Dockerfile"]:::docker
    click D "https://github.com/mahdi-dcs/ms-genomics-blog/tree/main/Dockerfile"
    D -->|"BuildsImage"| BE
    BE -->|"PullsImage"| BC

    E["submit_job.py"]:::job
    click E "https://github.com/mahdi-dcs/ms-genomics-blog/blob/main/submit_job.py"
    E -->|"SubmitsJob"| BC

    %% Nextflow Pipeline Layer
    subgraph "Nextflow Pipeline"
        F["Nextflow Main Pipeline"]:::pipeline
        click F "https://github.com/mahdi-dcs/ms-genomics-blog/blob/main/nextflow/main.nf"
        F -->|"Includes"| G["Modules"]:::pipeline
        click G "https://github.com/mahdi-dcs/ms-genomics-blog/tree/main/nextflow/modules"
        F -->|"Includes"| H["Subworkflows"]:::pipeline
        click H "https://github.com/mahdi-dcs/ms-genomics-blog/tree/main/nextflow/subworkflows"
        F -->|"Uses"| I["Nextflow Config"]:::pipeline
        click I "https://github.com/mahdi-dcs/ms-genomics-blog/blob/main/nextflow/nextflow.config"
        F -->|"Uses"| J["Run Nextflow"]:::pipeline
        click J "https://github.com/mahdi-dcs/ms-genomics-blog/blob/main/nextflow/run_nextflow.py"
    end

    %% Data and References
    K["Samples"]:::externalData
    click K "https://github.com/mahdi-dcs/ms-genomics-blog/tree/main/samples"
    L["Nextflow References"]:::externalData
    click L "https://github.com/mahdi-dcs/ms-genomics-blog/tree/main/nextflow-references"

    K -->|"UploadsData"| BB
    L -->|"UploadsData"| BA
    BB -->|"ProvidesData"| F
    BA -->|"ProvidesCode"| F

    %% Batch Pools interaction with Key Vault
    BC -->|"AuthenticatesUsing"| BD

    %% CI/CD trigger Nextflow Pipeline via Job Submission
    E -->|"Triggers"| F

    %% Styles
    classDef cicd fill:#FFA500,stroke:#333,stroke-width:2px;
    classDef bicep fill:#ADD8E6,stroke:#333,stroke-width:2px;
    classDef storage fill:#90EE90,stroke:#333,stroke-width:2px;
    classDef batch fill:#FF7F50,stroke:#333,stroke-width:2px;
    classDef kv fill:#FFB6C1,stroke:#333,stroke-width:2px;
    classDef acr fill:#FFD700,stroke:#333,stroke-width:2px;
    classDef pipeline fill:#DDA0DD,stroke:#333,stroke-width:2px;
    classDef docker fill:#F0E68C,stroke:#333,stroke-width:2px;
    classDef job fill:#F4A460,stroke:#333,stroke-width:2px;
    classDef externalData fill:#98FB98,stroke:#333,stroke-width:2px;
```
