How johnpatramanis/proteomic_pipeline works
• Dataset Initialization
• Dataset Construction
• Dataset Analysis
Each of these modules is implemented as its own “pipeline” that is orchestrated by Snakemake and leverages Conda environments to manage dependencies. There is also a GitHub_Tutorial section that helps users and developers understand how to set up and extend the workflows.
Main Components
• Dataset Initialization: Fetches reference protein/proteome data and other foundational resources.
• Dataset Construction: Deals with translation of protein data from whole-genome datasets (handling BAM, CRAM, and VCF inputs) and has output directories organized into formats (BAM, FASTA, gene-specific FASTAs, final output per protein/sample).
• Dataset Analysis: Handles alignment, formatting, and construction of phylogenetic trees, including both per-protein analyses and concatenated analyses.
• Workflow Engine: Represent Snakemake as the pipeline orchestrator – it controls the data flow between rules defined via Snakefiles in each module.
• Environment & Dependency Management: Use Conda environments (specified with YAML files) as a component ensuring reproducibility.
• Supplementary Tools and Scripts: Include R and Python scripts present in each module, which perform specific tasks (e.g., merging datasets, label fixing, assembly location fetching, and others).
• Data Repositories and Workspace: Show that each module maintains its own "Workspace" directory structure where intermediate and final files (e.g., FASTA files, VCF files, output of gene extraction, etc.) are stored.
• User Extensions and GitHub Tutorials: Optionally, include an “Extension/Integration” component based on the GitHub_Tutorial folder that provides guidance, extra data (e.g., sample lists) and supplementary information.
Relationships and Interactions
• That Dataset Construction further processes the data (downloaded BAM/VCF files, gene FASTA files) and prepares the final formatted sets.
• Then, Dataset Analysis takes the formatted sequences to perform alignment and phylogenetic tree construction.
• Integration of Conda environments used to isolate dependencies for specific Snakemake workflow executions.
Architectural Patterns and Design Principles
• Modular nature of the system — each pipeline is an independent yet inherently synergetic component.
• Use of a workflow orchestrator (Snakemake) to manage dependencies and parallel execution.
• How versioning and reproducibility is guaranteed through environment files (YAML) combined with structured workspace directories.
• That elements of pipelining (e.g., sequential data processing stages from raw input to final analysis output) are central to this system.
Technologies, Frameworks, and Libraries
• Snakemake: For orchestrating the workflow via Snakefiles found in each module.
• Conda: For managing software dependencies, indicated by the YAML files (Analyser.yml, Initiator.yml, Translator.yml, etc.).
• Scripting Languages: R (various R scripts) and Python (various Python scripts) – used to process, analyze, and format data.
• Bioinformatics Tools: Tools and packages (referenced in the supplementary PDF and tutorial) like MrBayes, RevBayes, BLAST, etc., should be noted as external dependencies handled by the Conda environments.
Specific Elements
• “Reference Data & Initialization” to “Dataset Construction” (which then processes BAM, VCF, FASTA, etc.)
• “Dataset Construction” to “Dataset Analysis” (which handles alignment and tree reconstruction)
• Optionally, add a note about extensibility where users can alter and extend these pipelines (highlight the “User Extensions” nature).