How tjpel/proteomicsanalysispipeline works
Project Type and Purpose
• This is a Python-based, configuration-driven data-processing pipeline (ETL/analysis tool) for proteomics data.
• It runs as a single command-line script (analysis_pipeline.py), reads raw Excel data, applies a user-specified sequence of cleaning, normalization, transformation, and statistical comparison steps, and emits both tabular and visualization outputs.
Main Architectural Components
a. Input Layer
• Raw data files (.xlsx) in {project_dir}/data/raw
• Example datasets under examples/…
b. Configuration Layer
• config.json: defines project information, pipeline steps (ordered_pipeline), comparisons, visualization and debug behaviors.
c. Orchestration Layer
• analysis_pipeline.py: entry point—parses config, orchestrates the execution sequence.
d. Core Processing Modules (helper_scripts/)
• data_operations.py: implements cleaning, missing-value removal, normalization, imputation, transformation, statistical computations (fold-change, p-value, group means).
• visualizations.py: generates PCA, volcano, and heatmap figures using Plotly (and optional Kaleido export).
• debug_wrapper.py: optional verbose logging and function-timing wrappers.
e. External Dependencies
• Python 3.12+, Pandas, NumPy, SciPy, Plotly, Kaleido, etc.
f. Data Stores
• Intermediate CSVs in {project_dir}/data/intermediate/ (no_processing.csv, pre_transformation.csv, post_pipeline.csv, fold_change_dataset.csv, dropped_proteins_list.csv, sample_info.csv).
• Final Excel output in {project_dir}/delivery/data/Data_With_Analysis.xlsx (multiple sheets: processed data, sample info, comparisons, removed proteins).
• Visualization files in {project_dir}/delivery/visualizations/ (PNG/JPG/PDF/SVG per comparison).
Relationships & Data Flows
1. User invokes analysis_pipeline.py → script loads config.json.
2. Orchestrator calls data-operations to:
• Read raw .xlsx → write no_processing.csv.
• Perform each configured pipeline step → generate pre_transformation.csv (if applicable) and post_pipeline.csv.
• Compute comparisons → write fold_change_dataset.csv, assemble final Excel sheets (including “Removed Proteins”).
3. Orchestrator calls visualization module → reads intermediate/final data → generates PCA, volcano, heatmap figures → writes to delivery path (and optionally opens in browser).
4. Debug wrapper intercepts calls for logging/timing if enabled.
Architectural Patterns & Principles
• Pipeline (ETL) pattern: each stage transforms data and writes to a shared filesystem location.
• Modular separation of concerns: orchestration, business logic (data ops), visualization, configuration, and debugging are isolated.
• Configuration-driven behavior: users control the pipeline without code changes.
• Filesystem as message bus: intermediate CSVs serve as “hand-off” between stages.