How angione-lab/nsclc-survival-prediction-models works
Project Type and Purpose
• This is a research/experimental deep-learning project focused on survival prediction for non-small cell lung cancer (NSCLC). It integrates radiological imaging, gene expression, and clinical data.
• The project implements pathway-guided models (H-VAE-Cox and XAT-VAE-Cox) to produce prognostic indexes, and it includes additional steps like SHAP for model interpretability as well as biological pathway analysis using KEGG and Reactome.
• The code is modular, supporting preprocessing (e.g., U-Net segmentation), model training/inference, and post-processing (visualization and statistical tests performed in R).
Analyzing the File Structure
• Top-Level Directories:
• Data: Contains raw and preprocessed data (CT scans, gene expression, clinical datasets, pathway masks, etc.).
• R: Includes R scripts for generating final plots (bar graphs, Kaplan-Meier curves, etc.) used for biological interpretation.
• Results: Hosts output files from different model runs and analyses.
• baselineModels: Contains reference implementations for baseline survival prediction models (DeepCoxMixture, pycox).
• models: Contains the deep learning model implementations (e.g., HVAECoxmodel, XVAECoxmodel, UnetModel, etc.).
• run: Contains scripts for end-to-end execution including data preprocessing (e.g., U-Net segmentation), feature generation for genes and images, survival prediction models, and SHAP interpretation.
• utils: Includes helper functions for data loading, image processing, survival analysis, and plotting.
• Config Files and Environments:
• Requirements.txt covers environment dependencies.
• .spyproject files indicate this has been developed in Spyder (Python IDE), and although they are IDE configuration files, they hint at the development practices.
• Architectural Patterns:
• The project is modularized where distinct directories represent different stages of the workflow (data ingestion, preprocessing, model training/inference, results, and plotting).
• There’s a separation between research baseline implementations (in baselineModels) and proposed model implementations (in models), which indicates layered experimentation.
a. Main Components:
• Data Layer:
• Data sources (raw CT scan images, gene expression datasets, clinical data, pathway annotations).
• External data feeds (links provided in the README and instructions on how to populate the Data folder).
• Preprocessing Layer:
• Image processing using U-Net (run/U-net-ImageSegmentation.py).
• Data cleaning, normalization, and feature selection (performed within the preprocessing scripts in run and utils).
• Feature Extraction / Dimensionality Reduction:
• Gene latent feature extraction (run/GenerateGeneLatentFeatures.py).
• Image latent feature extraction (run/GenerateImageLatentFeatures.py).
• Model Training and Inference:
• Deep learning survival prediction models including H-VAE-Cox and XAT-VAE-Cox (located in models and executed via the run scripts).
• Baseline models (found under baselineModels) for comparative analyses.
• Model Interpretation:
• SHAP analysis for model interpretation (run/H_VAE_Shap_Interpretation.py).
• Additional visualization and interpretation using Grad-CAM.
• Evaluation and Visualization:
• Survival analysis and calculation of the C-index, Kaplan-Meier curves (results and plotting scripts using R).
• Final plots and statistical results stored in the Results folder.
• Supporting Utilities:
• Utility functions under utils for loading data, handling images, and computing survival metrics.
b. Relationships and Interactions:
• Interactions between components, for example:
• The output from the U-Net image segmentation feeds into the latent feature extraction.
• Latent features from both gene expression and imaging data are then combined with clinical data to perform survival prediction.
• Model outputs are then used by the interpretation modules (SHAP) and further visualized using R scripts.
• For baseline models, indicate a parallel flow for comparison purposes alongside the main H-VAE-Cox and XAT-VAE-Cox approaches.
c. Architectural Patterns / Design Principles:
• Modularity: Separation into discrete, loosely coupled modules (Data, Preprocessing, Models, Post-Processing) allows for individual upgrades and testing.
• Pipeline/Workflow design: Data flows successively through a series of transformation stages (preprocessing → feature generation → prediction → interpretation).
• Reusability: Utility modules and baseline models indicate an emphasis on reusable components.
• Integration of heterogeneous data types: The architecture accommodates multiple data modalities (images, gene expression, and clinical data) unified by the deep learning models.
d. Relevant Technologies and Frameworks:
• Python-based deep learning frameworks (likely using TensorFlow/Keras or PyTorch) indicated by .h5 model files and script naming conventions.
• R for specialized statistical analysis and plots.
• External libraries for survival analysis, SHAP explanations, and image segmentation.
• U-Net architecture for image segmentation and autoencoder-based models for feature extraction.