# joan-smith/comprehensive-tcga-survival architecture

> Architecture diagram of the GitHub repository joan-smith/comprehensive-tcga-survival, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/joan-smith/comprehensive-tcga-survival
- Repository: https://github.com/joan-smith/comprehensive-tcga-survival

## Overview

##### Project type and purpose
- Type: A command-line, data-analysis pipeline in Python (not a web app).
- Purpose: Download TCGA data, run large-scale survival analyses by data type (CNV, mutations, methylation, miRNA, RNA-seq, RPPA), perform multivariate modeling and z-score calculations, then generate publication figures.

##### Main components
a. Data Sources & Download
– External: TCGA/GDC REST API or FTP
– Module: `comprehensive_tcga_survival/download_data.py`
– Output: Raw data files (~5 GiB) in a local data directory
b. Compute Environment
– HPC cluster or multi-core server (≥ 30 cores, hundreds of GB RAM)
– Conda or virtualenv environment (managed via `setup.py`)
c. Analysis Pipeline
– Orchestrator: `comprehensive_tcga_survival/main.py`
– Sub-modules for each data type:
- CNV: `cn.py`
- Mutations: `mutations.py`
- Methylation: `methylation.py` + gene sets
- miRNA: `mirna.py`
- RNA-seq: `rnaseq.py`
- RPPA: `rppa.py`
– Utilities:
- `zscore_common.py` (standardization)
- `stage_grade_parser.py` (clinical metadata)
– Multivariate modeling: `multivariate.py`
d. Figure Generation
– Scripts in `figs_and_figdata/` (e.g., kmplots.py, driver_genes.py)
– Input: Processed results (CSV, serialized objects)
– Output: Figures & tables for publication
e. Testing & CI
– Unit tests: `tests/`
– CI pipeline: `.github/workflows/pythonapp.yml`
– Test data under `tests/test_data/`

##### Relationships and data flow
- “Main Orchestrator” reads raw data, spawns parallel jobs (via multiprocessing or a scheduler).
- Each data-type module reads raw data → produces intermediate results.
- Utilities like z-score common functions are invoked by data-type modules.
- Multivariate module consumes per-datatype results → joint models.
- Figure scripts consume all final results → produce plots.
- Testing harness reads sample test data, runs modules, asserts outputs.

##### Architectural patterns and principles
– Modular design: one module per data type, reusable utility functions
– Parallelism: heavy compute offloaded to multi-core/HPC
– Configuration-by-convention: default data paths in scripts can be overridden

##### Technologies and frameworks
– Python 3.x
– Pandas, NumPy, SciPy, Statsmodels (implied dependencies)
– pytest for testing
– Conda for environment management
– GitHub Actions for CI

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion
    A["External TCGA/GDC API/FTP"]:::io
    B["Download Module\ndownload_data.py"]:::io
    C[(Raw Data Store)]:::io

    A -->|fetch data| B
    B -->|save raw files| C

    %% Compute Environment
    D["Environment Setup\nsetup.py"]:::compute
    C -->|data available| D

    %% Analysis Pipeline
    subgraph "Analysis Orchestrator"
        direction TB
        E["Main Orchestrator\nmain.py"]:::compute
    end
    D --> E

    subgraph "Data-Type Modules"
        direction TB
        F["CNV Analysis\ncn.py"]:::compute
        G["Mutations Analysis\nmutations.py"]:::compute
        H["Methylation Analysis\nmethylation.py"]:::compute
        I["miRNA Analysis\nmirna.py"]:::compute
        J["RNA-seq Analysis\nrnaseq.py"]:::compute
        K["RPPA Analysis\nrppa.py"]:::compute
    end

    E -->|parallel processing| F
    E -->|parallel processing| G
    E -->|parallel processing| H
    E -->|parallel processing| I
    E -->|parallel processing| J
    E -->|parallel processing| K

    %% Utilities
    subgraph Utilities
        direction TB
        U1["Z-score Utils\nzscore_common.py"]:::util
        U2["Stage/Grade Parser\nstage_grade_parser.py"]:::util
    end

    F -->|uses| U1
    G -->|uses| U1
    H -->|uses| U1
    I -->|uses| U1
    J -->|uses| U1
    K -->|uses| U1

    F -->|uses| U2
    G -->|uses| U2
    H -->|uses| U2
    I -->|uses| U2
    J -->|uses| U2
    K -->|uses| U2

    %% Multivariate Modeling
    L["Multivariate Modeling\nmultivariate.py"]:::compute
    F --> L
    G --> L
    H --> L
    I --> L
    J --> L
    K --> L

    L --> M[(Results Store)]:::io

    %% Figure Generation
    subgraph "Figures & Tables"
        direction TB
        N["Figure Scripts\nfigs_and_figdata/"]:::viz
    end
    M -->|input results| N
    N -->|output plots/tables| O["Publication Figures"]:::viz

    %% Testing & CI
    subgraph "Testing & CI"
        direction TB
        P["Unit Tests\ntests/"]:::test
        Q["Test Data\ntests/test_data/"]:::test
        R["CI Workflow\n.github/workflows/pythonapp.yml"]:::test
    end
    P --> E
    P --> F
    P --> G
    P --> H
    P --> I
    P --> J
    P --> K
    P --> U1
    P --> U2
    P --> L
    Q --> P
    R --> P

    %% Click events
    click B "https://github.com/joan-smith/comprehensive-tcga-survival/blob/main/comprehensive_tcga_survival/download_data.py"
    click D "https://github.com/joan-smith/comprehensive-tcga-survival/blob/main/setup.py"
    click R "https://github.com/joan-smith/comprehensive-tcga-survival/blob/main/.github/workflows/pythonapp.yml"
    click E "https://github.com/joan-smith/comprehensive-tcga-survival/blob/main/comprehensive_tcga_survival/main.py"
    click F "https://github.com/joan-smith/comprehensive-tcga-survival/blob/main/comprehensive_tcga_survival/cn.py"
    click G "https://github.com/joan-smith/comprehensive-tcga-survival/blob/main/comprehensive_tcga_survival/mutations.py"
    click H "https://github.com/joan-smith/comprehensive-tcga-survival/blob/main/comprehensive_tcga_survival/methylation.py"
    click I "https://github.com/joan-smith/comprehensive-tcga-survival/blob/main/comprehensive_tcga_survival/mirna.py"
    click J "https://github.com/joan-smith/comprehensive-tcga-survival/blob/main/comprehensive_tcga_survival/rnaseq.py"
    click K "https://github.com/joan-smith/comprehensive-tcga-survival/blob/main/comprehensive_tcga_survival/rppa.py"
    click U1 "https://github.com/joan-smith/comprehensive-tcga-survival/blob/main/comprehensive_tcga_survival/zscore_common.py"
    click U2 "https://github.com/joan-smith/comprehensive-tcga-survival/blob/main/comprehensive_tcga_survival/stage_grade_parser.py"
    click L "https://github.com/joan-smith/comprehensive-tcga-survival/blob/main/comprehensive_tcga_survival/multivariate.py"
    click N "https://github.com/joan-smith/comprehensive-tcga-survival/tree/main/figs_and_figdata/"
    click P "https://github.com/joan-smith/comprehensive-tcga-survival/tree/main/tests/"
    click Q "https://github.com/joan-smith/comprehensive-tcga-survival/tree/main/tests/test_data/"

    %% Styles
    classDef io fill:#ccffcc,stroke:#333,stroke-width:1px
    classDef compute fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef util fill:#ffe5cc,stroke:#333,stroke-width:1px
    classDef viz fill:#e5ccff,stroke:#333,stroke-width:1px
    classDef test fill:#dddddd,stroke:#333,stroke-width:1px
```
