# tjpel/proteomicsanalysispipeline architecture

> Architecture diagram of the GitHub repository tjpel/proteomicsanalysispipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tjpel/proteomicsanalysispipeline
- Repository: https://github.com/tjpel/proteomicsanalysispipeline

## Overview

##### Project Type and Purpose
- This is a Python-based, configuration-driven data-processing pipeline (ETL/analysis tool) for proteomics data.
- It runs as a single command-line script (`analysis_pipeline.py`), reads raw Excel data, applies a user-specified sequence of cleaning, normalization, transformation, and statistical comparison steps, and emits both tabular and visualization outputs.

##### Main Architectural Components
a. Input Layer
- Raw data files (`.xlsx`) in `{project_dir}/data/raw`
- Example datasets under `examples/…`
b. Configuration Layer
- `config.json`: defines project information, pipeline steps (`ordered_pipeline`), comparisons, visualization and debug behaviors.
c. Orchestration Layer
- `analysis_pipeline.py`: entry point—parses config, orchestrates the execution sequence.
d. Core Processing Modules (helper_scripts/)
- `data_operations.py`: implements cleaning, missing-value removal, normalization, imputation, transformation, statistical computations (fold-change, p-value, group means).
- `visualizations.py`: generates PCA, volcano, and heatmap figures using Plotly (and optional Kaleido export).
- `debug_wrapper.py`: optional verbose logging and function-timing wrappers.
e. External Dependencies
- Python 3.12+, Pandas, NumPy, SciPy, Plotly, Kaleido, etc.
f. Data Stores
- Intermediate CSVs in `{project_dir}/data/intermediate/` (no_processing.csv, pre_transformation.csv, post_pipeline.csv, fold_change_dataset.csv, dropped_proteins_list.csv, sample_info.csv).
- Final Excel output in `{project_dir}/delivery/data/Data_With_Analysis.xlsx` (multiple sheets: processed data, sample info, comparisons, removed proteins).
- Visualization files in `{project_dir}/delivery/visualizations/` (PNG/JPG/PDF/SVG per comparison).

##### Relationships & Data Flows
1. User invokes `analysis_pipeline.py` → script loads `config.json`.
2. Orchestrator calls data-operations to:
- Read raw `.xlsx` → write `no_processing.csv`.
- Perform each configured pipeline step → generate `pre_transformation.csv` (if applicable) and `post_pipeline.csv`.
- Compute comparisons → write `fold_change_dataset.csv`, assemble final Excel sheets (including “Removed Proteins”).
3. Orchestrator calls visualization module → reads intermediate/final data → generates PCA, volcano, heatmap figures → writes to delivery path (and optionally opens in browser).
4. Debug wrapper intercepts calls for logging/timing if enabled.

##### Architectural Patterns & Principles
- Pipeline (ETL) pattern: each stage transforms data and writes to a shared filesystem location.
- Modular separation of concerns: orchestration, business logic (data ops), visualization, configuration, and debugging are isolated.
- Configuration-driven behavior: users control the pipeline without code changes.
- Filesystem as message bus: intermediate CSVs serve as “hand-off” between stages.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Input Layer"
        RawExcel["Raw Excel Files"]:::file
    end

    subgraph "Configuration Layer"
        Config["config.json"]:::file
    end

    subgraph "Orchestration Layer"
        Orchestrator["analysis_pipeline.py"]:::code
    end

    subgraph "Processing Layer"
        DataOps["Data Operations Module"]:::code
        Visual["Visualization Module"]:::code
        Debug["Debug Wrapper Module"]:::code
    end

    subgraph "Intermediate Data Stores"
        NoProc["no_processing.csv"]:::datastore
        PreTrans["pre_transformation.csv"]:::datastore
        PostPipe["post_pipeline.csv"]:::datastore
        Fold["fold_change_dataset.csv"]:::datastore
        Dropped["dropped_proteins_list.csv"]:::datastore
        SampleInfo["sample_info.csv"]:::datastore
    end

    subgraph "Final Outputs"
        FinalExcel["Data_With_Analysis.xlsx"]:::datastore
        VizOut["Visualization Files"]:::datastore
    end

    subgraph "External Dependencies"
        Ext["Python Dependencies"]:::service
    end

    %% Flows
    RawExcel -->|reads raw data| DataOps
    Config -->|provides settings| Orchestrator
    Orchestrator -->|orchestrates| DataOps
    Orchestrator -->|orchestrates| Visual
    Orchestrator -->|wraps calls| Debug
    Debug -->|wraps| DataOps
    Debug -->|wraps| Visual

    DataOps -->|writes| NoProc
    DataOps -->|writes| PreTrans
    DataOps -->|writes| PostPipe
    DataOps -->|writes| Fold
    DataOps -->|writes| Dropped
    DataOps -->|writes| SampleInfo

    Visual -->|reads intermediates| PostPipe
    Visual -->|reads intermediates| Fold
    Visual -->|writes figures| VizOut

    DataOps -->|writes final sheets| FinalExcel

    Ext -.-> DataOps
    Ext -.-> Visual
    Ext -.-> Orchestrator

    %% Click Events
    click RawExcel "https://github.com/tjpel/proteomicsanalysispipeline/tree/main/examples/*/data/raw"
    click Config "https://github.com/tjpel/proteomicsanalysispipeline/blob/main/analysis_pipeline/config.json"
    click Orchestrator "https://github.com/tjpel/proteomicsanalysispipeline/blob/main/analysis_pipeline/analysis_pipeline.py"
    click DataOps "https://github.com/tjpel/proteomicsanalysispipeline/blob/main/analysis_pipeline/helper_scripts/data_operations.py"
    click Visual "https://github.com/tjpel/proteomicsanalysispipeline/blob/main/analysis_pipeline/helper_scripts/visualizations.py"
    click Debug "https://github.com/tjpel/proteomicsanalysispipeline/blob/main/analysis_pipeline/helper_scripts/debug_wrapper.py"
    click Ext "https://github.com/tjpel/proteomicsanalysispipeline/blob/main/analysis_pipeline/requirements.txt"
    click NoProc "https://github.com/tjpel/proteomicsanalysispipeline/tree/main/examples/*/data/intermediate"
    click PreTrans "https://github.com/tjpel/proteomicsanalysispipeline/tree/main/examples/*/data/intermediate"
    click PostPipe "https://github.com/tjpel/proteomicsanalysispipeline/tree/main/examples/*/data/intermediate"
    click Fold "https://github.com/tjpel/proteomicsanalysispipeline/tree/main/examples/*/data/intermediate"
    click Dropped "https://github.com/tjpel/proteomicsanalysispipeline/tree/main/examples/*/data/intermediate"
    click SampleInfo "https://github.com/tjpel/proteomicsanalysispipeline/tree/main/examples/*/data/intermediate"
    click FinalExcel "https://github.com/tjpel/proteomicsanalysispipeline/blob/main/examples/*/delivery/data/Data_With_Analysis.xlsx"
    click VizOut "https://github.com/tjpel/proteomicsanalysispipeline/tree/main/examples/*/delivery/visualizations/"

    %% Styles
    classDef code fill:#ADD8E6,stroke:#000;
    classDef datastore fill:#90EE90,stroke:#000;
    classDef file fill:#FFA500,stroke:#000;
    classDef service fill:#D3D3D3,stroke:#000;
```
