# johnpatramanis/proteomic_pipeline architecture

> Architecture diagram of the GitHub repository johnpatramanis/proteomic_pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/johnpatramanis/proteomic_pipeline
- Repository: https://github.com/johnpatramanis/proteomic_pipeline

## Overview

- Dataset Initialization
- Dataset Construction
- Dataset Analysis

Each of these modules is implemented as its own “pipeline” that is orchestrated by Snakemake and leverages Conda environments to manage dependencies. There is also a GitHub_Tutorial section that helps users and developers understand how to set up and extend the workflows.

##### Main Components
- Dataset Initialization: Fetches reference protein/proteome data and other foundational resources.
- Dataset Construction: Deals with translation of protein data from whole-genome datasets (handling BAM, CRAM, and VCF inputs) and has output directories organized into formats (BAM, FASTA, gene-specific FASTAs, final output per protein/sample).
- Dataset Analysis: Handles alignment, formatting, and construction of phylogenetic trees, including both per-protein analyses and concatenated analyses.
- Workflow Engine: Represent Snakemake as the pipeline orchestrator – it controls the data flow between rules defined via Snakefiles in each module.
- Environment & Dependency Management: Use Conda environments (specified with YAML files) as a component ensuring reproducibility.
- Supplementary Tools and Scripts: Include R and Python scripts present in each module, which perform specific tasks (e.g., merging datasets, label fixing, assembly location fetching, and others).
- Data Repositories and Workspace: Show that each module maintains its own "Workspace" directory structure where intermediate and final files (e.g., FASTA files, VCF files, output of gene extraction, etc.) are stored.
- User Extensions and GitHub Tutorials: Optionally, include an “Extension/Integration” component based on the GitHub_Tutorial folder that provides guidance, extra data (e.g., sample lists) and supplementary information.

##### Relationships and Interactions
- That Dataset Construction further processes the data (downloaded BAM/VCF files, gene FASTA files) and prepares the final formatted sets.
- Then, Dataset Analysis takes the formatted sequences to perform alignment and phylogenetic tree construction.
- Integration of Conda environments used to isolate dependencies for specific Snakemake workflow executions.

##### Architectural Patterns and Design Principles
- Modular nature of the system — each pipeline is an independent yet inherently synergetic component.
- Use of a workflow orchestrator (Snakemake) to manage dependencies and parallel execution.
- How versioning and reproducibility is guaranteed through environment files (YAML) combined with structured workspace directories.
- That elements of pipelining (e.g., sequential data processing stages from raw input to final analysis output) are central to this system.

##### Technologies, Frameworks, and Libraries
- Snakemake: For orchestrating the workflow via Snakefiles found in each module.
- Conda: For managing software dependencies, indicated by the YAML files (Analyser.yml, Initiator.yml, Translator.yml, etc.).
- Scripting Languages: R (various R scripts) and Python (various Python scripts) – used to process, analyze, and format data.
- Bioinformatics Tools: Tools and packages (referenced in the supplementary PDF and tutorial) like MrBayes, RevBayes, BLAST, etc., should be noted as external dependencies handled by the Conda environments.

##### Specific Elements
- “Reference Data & Initialization” to “Dataset Construction” (which then processes BAM, VCF, FASTA, etc.)
- “Dataset Construction” to “Dataset Analysis” (which handles alignment and tree reconstruction)
- Optionally, add a note about extensibility where users can alter and extend these pipelines (highlight the “User Extensions” nature).

## Diagram (Mermaid)

```mermaid
flowchart TD
    Conda["Conda Environments"]:::conda
    Snakemake["Snakemake Orchestrator"]:::orchestrator
    Tutorial["GitHub Tutorial"]:::tutorial

    subgraph "Dataset Initialization Pipeline"
        DI_YAML["Initiator YAML"]:::yaml
        DI_Snake["Snakefile"]:::snake
        DI_Python["Python Scripts"]:::script
        DI_Workspace["Workspace"]:::workspace

        DI_YAML --> DI_Snake
        DI_Snake --> DI_Python
        DI_Snake --> DI_Workspace
    end

    subgraph "Dataset Construction Pipeline"
        DC_YAML["Translator YAML"]:::yaml
        DC_Snake["Snakefile"]:::snake
        DC_Python["Python Scripts"]:::script
        DC_R["R Scripts"]:::script
        DC_Genes["Genes Data"]:::data
        DC_EIT["EIT Data"]:::data
        DC_Workspace["Workspace"]:::workspace

        DC_YAML --> DC_Snake
        DC_Snake --> DC_Python
        DC_Snake --> DC_R
        DC_Snake --> DC_Genes
        DC_Snake --> DC_EIT
        DC_Genes --> DC_Workspace
        DC_EIT --> DC_Workspace
    end

    subgraph "Dataset Analysis Pipeline"
        DA_YAML["Analyser YAML"]:::yaml
        DA_Snake["Snakefile"]:::snake
        DA_R["R Scripts"]:::script
        DA_Workspace["Workspace"]:::workspace

        DA_YAML --> DA_Snake
        DA_Snake --> DA_R
        DA_R --> DA_Workspace
    end

    %% Connect Conda to YAML configuration files
    Conda --> DI_YAML
    Conda --> DC_YAML
    Conda --> DA_YAML

    %% Connect Snakemake Orchestrator to all Snakefiles
    Snakemake --> DI_Snake
    Snakemake --> DC_Snake
    Snakemake --> DA_Snake

    %% Data flow between pipelines
    DI_Workspace -->|"init_outputs"| DC_Snake
    DC_Workspace -->|"constructed_data"| DA_Snake

    %% GitHub Tutorial as a resource for user extensions
    Tutorial -.-> Snakemake

    %% Styles
    classDef yaml fill:#FAD7A0,stroke:#D35400,stroke-width:2px;
    classDef snake fill:#A9DFBF,stroke:#196F3D,stroke-width:2px;
    classDef script fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef data fill:#F5CBA7,stroke:#D35400,stroke-width:2px;
    classDef workspace fill:#D2B4DE,stroke:#884EA0,stroke-width:2px;
    classDef orchestrator fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef conda fill:#FCF3CF,stroke:#B7950B,stroke-width:2px;
    classDef tutorial fill:#D7BDE2,stroke:#8E44AD,stroke-width:2px;

    %% Click Events for Dataset Initialization Pipeline
    click DI_YAML "https://github.com/johnpatramanis/proteomic_pipeline/blob/main/Dataset_Initialization/Initiator.yml"
    click DI_Snake "https://github.com/johnpatramanis/proteomic_pipeline/tree/main/Dataset_Initialization/Snakefile"
    click DI_Python "https://github.com/johnpatramanis/proteomic_pipeline/tree/main/Dataset_Initialization/Python_Scripts/"
    click DI_Workspace "https://github.com/johnpatramanis/proteomic_pipeline/tree/main/Dataset_Initialization/Workspace/"

    %% Click Events for Dataset Construction Pipeline
    click DC_YAML "https://github.com/johnpatramanis/proteomic_pipeline/blob/main/Dataset_Construction/Translator.yml"
    click DC_Snake "https://github.com/johnpatramanis/proteomic_pipeline/tree/main/Dataset_Construction/Snakefile"
    click DC_Python "https://github.com/johnpatramanis/proteomic_pipeline/tree/main/Dataset_Construction/Python Scripts/"
    click DC_R "https://github.com/johnpatramanis/proteomic_pipeline/tree/main/Dataset_Construction/R scripts/"
    click DC_Genes "https://github.com/johnpatramanis/proteomic_pipeline/tree/main/Dataset_Construction/Genes/"
    click DC_EIT "https://github.com/johnpatramanis/proteomic_pipeline/tree/main/Dataset_Construction/EIT/"
    click DC_Workspace "https://github.com/johnpatramanis/proteomic_pipeline/tree/main/Dataset_Construction/Workspace/"

    %% Click Events for Dataset Analysis Pipeline
    click DA_YAML "https://github.com/johnpatramanis/proteomic_pipeline/blob/main/Dataset_Analysis/Analyser.yml"
    click DA_Snake "https://github.com/johnpatramanis/proteomic_pipeline/tree/main/Dataset_Analysis/Snakefile"
    click DA_R "https://github.com/johnpatramanis/proteomic_pipeline/tree/main/Dataset_Analysis/Rscripts/"
    click DA_Workspace "https://github.com/johnpatramanis/proteomic_pipeline/tree/main/Dataset_Analysis/Workspace/"

    %% Click Event for GitHub Tutorial
    click Tutorial "https://github.com/johnpatramanis/proteomic_pipeline/tree/main/GitHub_Tutorial/"
```
