# mahmoodlab/porpoise architecture

> Architecture diagram of the GitHub repository mahmoodlab/porpoise, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mahmoodlab/porpoise
- Repository: https://github.com/mahmoodlab/porpoise

## Overview

##### Project type and purpose
- This is a command-line driven, Python-based research pipeline for pan-cancer prognosis using multimodal deep learning.
- It integrates whole-slide image (WSI) patch features and molecular profile data to train and evaluate survival models across 14 cancer types.

##### Main components of the system
a. External Data Sources
- GDC Data Portal (WSI: .svs files)
- cBioPortal (molecular CSVs)
b. Preprocessing (CLAM toolbox)
- Tissue segmentation → 256×256 patches → ResNet50 encoding → .pt feature files
c. Data Storage
- DATA_ROOT_DIR/ (per-cancer subfolders of .pt files)
- datasets_csv/ & datasets_csv_mutsig/ (molecular feature CSVs)
- splits/5foldcv/ (train/val split CSVs)
d. Dataset Loader Layer
- datasets/dataset_generic.py & dataset_survival.py implement PyTorch Dataset classes that:
– ingest .pt features + molecular CSVs + splits
– yield (image_features, molecular_features, label) batches
e. Model Definitions
- models/model_set_mil.py (SNN / set-based MIL)
- models/model_coattn.py (co-attention MIL)
- models/model_genomic.py (genomic-only network)
- models/model_porpoise.py (multimodal fusion network)
- models/model_utils.py (shared layers)
f. Training & Evaluation Utilities
- utils/*.py (train loops, loss functions, regularization, file I/O, evaluation metrics)
- core_utils.py orchestrates experiment setup
g. Control Scripts
- main.py – parses CLI args (--mode, --model_type, --which_splits, etc.), launches training
- eval_surv.py – loads checkpoints, computes survival metrics and visualizations
h. Documentation
- docs/INSTALLATION.md, docs/Commands.md, docs/README.md

##### Relationships and data flow
1) Raw WSIs (.svs) & molecular CSVs → CLAM toolbox (external) → .pt feature files + cleaned CSVs stored locally
2) splits/5foldcv defines train/val Case IDs for each cancer type
3) Dataset classes load features + CSVs + split files → PyTorch DataLoader
4) main.py passes DataLoader → selected Model → utils functions drive training loop → produces:
– model checkpoints (.pt)
– tensorboard logs, evaluation CSVs
5) eval_surv.py consumes checkpoints + splits → generates final performance metrics

##### Architectural patterns and design principles
- Pipeline pattern: Preprocessing → Storage → Loading → Training → Evaluation
- Modular package structure (datasets, models, utils) for separation of concerns
- CLI-driven orchestration to swap models, modalities, splits without code changes
- Use of standard ML frameworks (PyTorch, TensorFlow, Captum, SHAP) abstracted behind utility modules

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Sources
    subgraph "External Data Sources"
        GDC["GDC Data Portal (.svs)"]:::external
        CBP["cBioPortal (CSV)"]:::external
    end

    %% Preprocessing
    PREP["CLAM Preprocessing"]:::process

    %% Data Storage
    subgraph "Data Storage"
        PT[".pt Files\n(DATA_ROOT_DIR)"]:::store
        MCSV["Molecular CSVs"]:::store
        MCSV2["MutSig CSVs"]:::store
        SPLITS["5-fold Splits"]:::store
    end

    %% Dataset Loader
    subgraph "Dataset Loader Layer"
        DG["dataset_generic.py\n(Generic MIL Dataset)"]:::code
        DS["dataset_survival.py\n(Survival Dataset)"]:::code
    end

    %% Model Suite
    subgraph "Model Suite"
        MSN["model_set_mil.py\n(Set-Based MIL)"]:::code
        MCA["model_coattn.py\n(Co-Attention MIL)"]:::code
        MGN["model_genomic.py\n(Genomic-Only)"]:::code
        MPO["model_porpoise.py\n(Porpoise Fusion)"]:::code
        MU["model_utils.py\n(Shared Layers)"]:::code
    end

    %% Training Engine
    subgraph "Training Engine"
        MAIN["main.py\n(CLI Training)"]:::script
        CTU["cluster_train_utils.py"]:::code
        CATU["coattn_train_utils.py"]:::code
        CORE["core_utils.py"]:::code
        FIO["file_utils.py"]:::code
        LOSS["loss_func.py"]:::code
        MMREG["mm_regularization.py"]:::code
        UTILS["utils.py"]:::code
    end

    %% Outputs
    subgraph "Outputs"
        CHECKPTS["Model Checkpoints (.pt)"]:::store
        LOGS["TensorBoard Logs"]:::store
        EVACSV["Evaluation CSVs"]:::store
    end

    %% Evaluation Engine
    subgraph "Evaluation Engine"
        EVALS["eval_surv.py\n(Survival Eval)"]:::script
        EVUTIL["eval_utils.py"]:::code
        FM["Final Performance Metrics"]:::store
    end

    %% Documentation
    subgraph "Documentation"
        DOC1["INSTALLATION.md"]:::doc
        DOC2["Commands.md"]:::doc
        DOC3["README.md"]:::doc
        CLAMCFG["clam.yaml"]:::doc
    end

    %% Relationships
    GDC -->|"svs → .pt"| PREP
    CBP -->|"raw CSV → cleaned CSV"| PREP

    PREP -->|"features → .pt"| PT
    PREP -->|"clean molecular CSVs"| MCSV
    PREP -->|"mutSig CSVs"| MCSV2

    PT --> DG
    MCSV --> DG
    MCSV2 --> DG
    SPLITS --> DG

    PT --> DS
    MCSV --> DS
    MCSV2 --> DS
    SPLITS --> DS

    DG --> MAIN
    DS --> MAIN

    MSN --> MAIN
    MCA --> MAIN
    MGN --> MAIN
    MPO --> MAIN
    MU --> MAIN

    CTU --> MAIN
    CATU --> MAIN
    CORE --> MAIN
    FIO --> MAIN
    LOSS --> MAIN
    MMREG --> MAIN
    UTILS --> MAIN

    MAIN --> CHECKPTS
    MAIN --> LOGS
    MAIN --> EVACSV

    CHECKPTS --> EVALS
    SPLITS --> EVALS
    EVUTIL --> EVALS
    EVALS --> FM

    %% Click Events
    click DG "https://github.com/mahmoodlab/porpoise/blob/master/datasets/dataset_generic.py"
    click DS "https://github.com/mahmoodlab/porpoise/blob/master/datasets/dataset_survival.py"
    click MCSV "https://github.com/mahmoodlab/porpoise/tree/master/datasets_csv/"
    click MCSV2 "https://github.com/mahmoodlab/porpoise/tree/master/datasets_csv_mutsig/"
    click SPLITS "https://github.com/mahmoodlab/porpoise/tree/master/splits/5foldcv/"
    click MSN "https://github.com/mahmoodlab/porpoise/blob/master/models/model_set_mil.py"
    click MCA "https://github.com/mahmoodlab/porpoise/blob/master/models/model_coattn.py"
    click MGN "https://github.com/mahmoodlab/porpoise/blob/master/models/model_genomic.py"
    click MPO "https://github.com/mahmoodlab/porpoise/blob/master/models/model_porpoise.py"
    click MU "https://github.com/mahmoodlab/porpoise/blob/master/models/model_utils.py"
    click CTU "https://github.com/mahmoodlab/porpoise/blob/master/utils/cluster_train_utils.py"
    click CATU "https://github.com/mahmoodlab/porpoise/blob/master/utils/coattn_train_utils.py"
    click CORE "https://github.com/mahmoodlab/porpoise/blob/master/utils/core_utils.py"
    click EVUTIL "https://github.com/mahmoodlab/porpoise/blob/master/utils/eval_utils.py"
    click FIO "https://github.com/mahmoodlab/porpoise/blob/master/utils/file_utils.py"
    click LOSS "https://github.com/mahmoodlab/porpoise/blob/master/utils/loss_func.py"
    click MMREG "https://github.com/mahmoodlab/porpoise/blob/master/utils/mm_regularization.py"
    click UTILS "https://github.com/mahmoodlab/porpoise/blob/master/utils/utils.py"
    click MAIN "https://github.com/mahmoodlab/porpoise/blob/master/main.py"
    click EVALS "https://github.com/mahmoodlab/porpoise/blob/master/eval_surv.py"
    click DOC1 "https://github.com/mahmoodlab/porpoise/blob/master/docs/INSTALLATION.md"
    click DOC2 "https://github.com/mahmoodlab/porpoise/blob/master/docs/Commands.md"
    click DOC3 "https://github.com/mahmoodlab/porpoise/blob/master/docs/README.md"
    click CLAMCFG "https://github.com/mahmoodlab/porpoise/blob/master/docs/clam.yaml"

    %% Styles
    classDef external fill:#cee,stroke:#036,stroke-width:1px
    classDef process fill:#f96,stroke:#963,stroke-width:1px
    classDef store fill:#9f6,stroke:#393,stroke-width:1px
    classDef code fill:#c9f,stroke:#639,stroke-width:1px
    classDef script fill:#ff9,stroke:#993,stroke-width:1px
    classDef doc fill:#ddd,stroke:#666,stroke-width:1px,stroke-dasharray: 5 5
```
