How joan-smith/comprehensive-tcga-survival works
Project type and purpose
• Type: A command-line, data-analysis pipeline in Python (not a web app).
• Purpose: Download TCGA data, run large-scale survival analyses by data type (CNV, mutations, methylation, miRNA, RNA-seq, RPPA), perform multivariate modeling and z-score calculations, then generate publication figures.
Main components
a. Data Sources & Download
– External: TCGA/GDC REST API or FTP
– Module: comprehensive_tcga_survival/download_data.py
– Output: Raw data files (~5 GiB) in a local data directory
b. Compute Environment
– HPC cluster or multi-core server (≥ 30 cores, hundreds of GB RAM)
– Conda or virtualenv environment (managed via setup.py)
c. Analysis Pipeline
– Orchestrator: comprehensive_tcga_survival/main.py
– Sub-modules for each data type:
• CNV: cn.py
• Mutations: mutations.py
• Methylation: methylation.py + gene sets
• miRNA: mirna.py
• RNA-seq: rnaseq.py
• RPPA: rppa.py
– Utilities:
• zscore_common.py (standardization)
• stage_grade_parser.py (clinical metadata)
– Multivariate modeling: multivariate.py
d. Figure Generation
– Scripts in figs_and_figdata/ (e.g., kmplots.py, driver_genes.py)
– Input: Processed results (CSV, serialized objects)
– Output: Figures & tables for publication
e. Testing & CI
– Unit tests: tests/
– CI pipeline: .github/workflows/pythonapp.yml
– Test data under tests/test_data/
Relationships and data flow
• “Main Orchestrator” reads raw data, spawns parallel jobs (via multiprocessing or a scheduler).
• Each data-type module reads raw data → produces intermediate results.
• Utilities like z-score common functions are invoked by data-type modules.
• Multivariate module consumes per-datatype results → joint models.
• Figure scripts consume all final results → produce plots.
• Testing harness reads sample test data, runs modules, asserts outputs.
Architectural patterns and principles
– Modular design: one module per data type, reusable utility functions
– Parallelism: heavy compute offloaded to multi-core/HPC
– Configuration-by-convention: default data paths in scripts can be overridden
Technologies and frameworks
– Python 3.x
– Pandas, NumPy, SciPy, Statsmodels (implied dependencies)
– pytest for testing
– Conda for environment management
– GitHub Actions for CI