How mahdi-dcs/ms-genomics-blog works
Main Components
• Infrastructure Deployment:
• Bicep Templates (located under the “infra” folder) for provisioning resources such as Batch Pools, Storage Accounts, and associated modules (batch, key-vault, roles, storage).
• Assumed external components (Virtual Network, Private Endpoints Subnet, DNS Zones) that are part of the Enterprise Landing Zone.
• Storage Accounts:
• One storage account for pipeline assets and source code (with versioning enabled, HNS disabled).
• One ADLS Gen2 account for Nextflow I/O files, supporting file-level access control.
• Compute Resources:
• Two Azure Batch Pools:
• “Nextflow Runners” (small VMs to run Nextflow commands)
• “Pipeline Runners” (larger VMs for individual pipeline tasks)
• Containerization:
• A Dockerfile builds a custom Nextflow image (using the official Nextflow image plus required Python libraries), ultimately pushed to an Azure Container Registry (ACR).
• Nextflow Pipeline:
• Main script (nextflow/main.nf) and a set of modules (in the “nextflow/modules” directory) for various tasks (alignment, variant calling, etc.).
• Subworkflows (in “nextflow/subworkflows”) to handle particular pipelines for CMV and HBV.
• Data and References:
• Sequencing samples are stored in the “samples” directory.
• Reference data (including FASTA files, blast databases, and other genomic resources) reside in the “nextflow-references” folder.
• Job Submission:
• A Python submission script (submit_job.py) to upload pipeline codes and submit a Batch job.
• CI/CD Pipeline:
• A GitHub Actions workflow (in .github/workflows/deploy-infrastructure.yaml) that handles deployment of the infrastructure using the Bicep templates.
Relationships and Interactions
• The Bicep templates deploy the core infrastructure, including storage accounts, Batch pools, and key vaults. These resources form the backbone of the deployment environment.
• The GitHub Actions workflow automates the deployment by triggering on code commits and executing the Bicep templates.
• The Dockerfile is used to build a custom container image that is then pushed to the ACR; this image is later utilized by the Nextflow runner nodes on the Batch pools.
• The Nextflow pipeline (with its main script and modular sub-scripts) is executed on the Batch pools. It interacts with:
• The I/O storage account to read sample data and reference files (uploaded from “samples” and “nextflow-references”).
• The assets storage account which holds pipeline code and possibly configuration for reproducibility.
• The submit_job.py script ties it together by orchestrating job submission; it ensures that the appropriate environment variables (pointing to storage accounts, Batch account, key vault, etc.) are set before triggering the pipeline.
Architectural Patterns and Design Principles
• Infrastructure as Code: The complete infrastructure is defined through Bicep templates (in the “infra” folder), ensuring repeatable and version-controlled deployments.
• Containerization: The use of a custom Docker image allows consistency across execution environments on Azure Batch.
• Separation of Concerns: Distinct components and data flows separate the pipeline execution (Nextflow) from infrastructure provisioning (Bicep) and CI/CD processes (GitHub Actions).
• Secure Access and Scalability: Managed identities, controlled access via Key Vault, and leveraging Azure Batch pools reflect an emphasis on secure, scalable, and manageable deployment.