# gevaertlab/sequoia-pub architecture

> Architecture diagram of the GitHub repository gevaertlab/sequoia-pub, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gevaertlab/sequoia-pub
- Repository: https://github.com/gevaertlab/sequoia-pub

## Overview

Below are the detailed guidelines and a proposed strategy for mapping the project’s architecture:

- Data Ingestion & Pre-Processing:
- Initial stage where WSIs are handled.
- “Patch Extraction” (pre_processing/patch_gen_hdf5.py and scripts/extract_patch.sh) and “Feature Extraction” (compute_features_hdf5.py under pre_processing and scripts/extract_resnet_features.sh).
- That after feature extraction the k-Means clustering is applied (via scripts/extract_kmean_features.sh). Emphasize that these phases transform raw WSIs into feature vectors (stored in HDF5 files).

- Model Training & Fine-tuning:
- Core training module located in the “src” folder. Components include:
o The main training driver (src/main.py).
o The model definitions (src/resnet.py, src/vit.py, src/tformer_lin.py, and src/he2rna.py for the MLP based HE2RNA variant).
o The pre-training script (src/pretrain_gtex.py) and fold definition files (src/folds) for cross-validation splits.
- That training can be run with various model types (vis for the SEQUOIA linearized transformer, vit for standard transformer aggregation, and he2rna for the MLP variant).

- Evaluation & Benchmarking:
- “evaluation” module including evaluation/predict_independent_dataset.py (for scoring an independent dataset) and evaluation/evaluate_model.py (for computing Pearson correlation, RMSE, and filtering significant genes).
- Gene list CSV (evaluation/gene_list.csv) as part of input/output, used to map predicted gene indexes to their names.
- That benchmarking entails comparing model predictions versus ground truth.

- Spatial Visualization:
- Separate the spatial analysis component (spatial_vis folder) that focuses on spatial gene expression mapping.

- Driver/Utility Scripts & Examples:
- “scripts” folder (for bash scripts) and “examples” folder that provides sample reference files.
- These components glue the modular pieces together and allow execution of the full pipeline.

##### Data Flow & Interactions
- From the initial pre-processing block:
- The output of pre-processing (the feature HDF5 files) is then fed into the Model Training/Fine-tuning block.
- Within Model Training:
- Use of multiple training scripts – one path for pre-training (on GTEx data) and one for fine-tuning on TCGA data.
- Dependency on external pre-trained weights loaded via HuggingFace (as noted in the README).
- Model Evaluation:
- Spatial Visualization:
- That outputs from the trained model also can be fed into the “Spatial Visualization” module to generate spatial gene expression maps.
- Additional computation pipelines like Earth Mover’s Distance (EMD) measurement.

##### Architectural Patterns and Design Principles
- Modularity: The system is broken into clearly separated folders that encapsulate distinct stages (pre-processing, training, evaluation, and visualization).
- Pipeline Architecture: Data flows sequentially from raw WSIs through pre-processing, feature extraction, model training, evaluation, and finally spatial visualization.
- External Dependencies: Emphasize where external libraries or interfaces come into play (Openslide for WSI processing, HuggingFace for model checkpoints, and dependency management via Conda and requirements.txt).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Dependencies
    OD["Openslide"]:::external
    HF["HuggingFace"]:::external

    %% Data Ingestion & Pre-Processing
    subgraph "Data Ingestion & Pre-Processing"
        subgraph "Patch Extraction"
            P1["patch_gen_hdf5.py"]:::preproc
            P2["extract_patch.sh"]:::preproc
        end
        subgraph "Feature Extraction"
            F1["compute_features_hdf5.py"]:::preproc
            F2["extract_resnet_features.sh"]:::preproc
        end
        subgraph "K-Means Clustering"
            K1["extract_kmean_features.sh"]:::preproc
            K2["kmean_features.py"]:::preproc
        end
    end

    %% Model Training & Fine-tuning
    subgraph "Model Training & Fine-tuning"
        MT["src/main.py"]:::training
        subgraph "Model Definitions"
            MD1["resnet.py"]:::training
            MD2["vit.py"]:::training
            MD3["tformer_lin.py"]:::training
            MD4["he2rna.py"]:::training
        end
        PT["src/pretrain_gtex.py"]:::training
        FOLD["src/folds"]:::training
    end

    %% Evaluation & Benchmarking
    subgraph "Evaluation & Benchmarking"
        subgraph "Evaluation Scripts"
            Eval1["evaluate_model.py"]:::evaluation
            Eval2["predict_independent_dataset.py"]:::evaluation
        end
        GeneCSV["gene_list.csv"]:::evaluation
        Corr["CorrelationStats.py"]:::evaluation
    end

    %% Spatial Visualization
    subgraph "Spatial Visualization"
        subgraph "Visualization Logic"
            Vis1["visualize.py"]:::visualization
            Vis2["get_emd.py"]:::visualization
            Vis3["gbm_celltype_analysis.py"]:::visualization
        end
    end

    %% Driver/Utility Scripts & Examples
    subgraph "Driver/Utility Scripts & Examples"
        subgraph "Utility Bash Scripts"
            Util1["run_train.sh"]:::utility
            Util2["run_he2rna.sh"]:::utility
            Util3["run_visualize.sh"]:::utility
        end
        ExRef["ref_file.csv"]:::utility
    end

    %% Data Flow Connections
    OD --> P1
    ExRef --> P1
    P2 --> F1
    F2 --> K1
    K2 --> MT
    MT --> MD1
    MT --> PT
    MT --> FOLD
    MT --> Eval1
    Eval2 --> GeneCSV
    Eval2 --> Corr
    MT --> Vis1
    Vis1 --> Vis2
    Vis2 --> Vis3
    Util1 --> MT
    HF --> PT

    %% Styles
    classDef preproc fill:#cce5ff,stroke:#007bff,color:#000,stroke-width:2px;
    classDef training fill:#d4edda,stroke:#28a745,color:#000,stroke-width:2px;
    classDef evaluation fill:#ffe5b4,stroke:#fd7e14,color:#000,stroke-width:2px;
    classDef visualization fill:#e2ccff,stroke:#6f42c1,color:#000,stroke-width:2px;
    classDef utility fill:#fff3cd,stroke:#ffc107,color:#000,stroke-width:2px;
    classDef external fill:#f8d7da,stroke:#dc3545,color:#000,stroke-width:2px;
```
