How jiagengchang/myevae works
Project Type and Purpose
• MyeVAE is a Python-based CLI tool and library implementing a multi-modal Variational Autoencoder for risk prediction in multiple myeloma patients.
• Core workflows: feature preprocessing, model training, scoring, and SHAP analysis—each invoked via myevae commands.
File-Structure Analysis
• Top-level:
• LICENSE, README.md, pyproject.toml, requirements.txt (project metadata & dependencies)
• assets/ (illustrations)
• src/myevae/ – code modules
• Key modules under src/myevae:
• __main__.py — CLI entrypoint parses subcommands (preprocess, train, score, shap)
• pipelinetools.py — orchestrates end-to-end pipelines per subcommand
• preprocess.py — feature selection (elastic net CoxPH), imputation (IterativeImputer)
• dataset.py — PyTorch Dataset/DataLoader wrappers for CSV inputs
• buildnetwork.py & nn.py — constructs the VAE encoder-decoder graph, including multi-omics subtasks
• coxphloss.py, kldivergence.py — custom loss components (Cox proportional hazards & KL divergence)
• train.py & estimator.py — training loops, early stopping, hyperparameter search via param_grid.py
• score.py — loads trained model, computes validation/test metrics
• shap.py & plotshap.py — invokes SHAP value computation and plotting
• param_grid.py — default hyperparameter grid for testing; override via CLI.
README Insights
• Four sequential CLI stages: preprocess → train → score → shap
• Inputs: raw or preprocessed CSVs + optional custom param_grid.py
• Outputs: processed CSVs, model weights (.pth), metrics (.json), SHAP plots
• Dependencies: Python ≥ 3.9, PyTorch, scikit-learn, scikit-survival, modified SHAP.
a. Main Components
1. CLI Layer (__main__.py)
2. Pipeline Orchestrator (pipelinetools.py)
3. Data Layer
• Raw CSV inputs
• preprocess.py → processed CSVs
• dataset.py → DataLoader
4. Model Layer
• buildnetwork.py + nn.py (VAE architecture)
• Loss modules (coxphloss.py, kldivergence.py)
5. Training & Evaluation
• estimator.py & train.py (fit, grid search, early stopping)
• score.py (metrics computation)
6. Explainability
• shap.py & plotshap.py (SHAP value generation, visualizations)
7. Configuration
• param_grid.py (hyperparams)
8. External Data Store (optional)
• AWS S3 buckets for example inputs/outputs
b. Relationships & Interactions
• CLI → Pipeline Orchestrator
• Orchestrator invokes either preprocessing, training, scoring or SHAP pipelines
• Preprocessing pipeline reads raw CSVs → writes processed CSVs
• Training pipeline reads processed CSVs + param_grid.py → builds network → trains model → writes .pth weights + metrics
• Scoring pipeline loads .pth → computes metrics → writes .json
• SHAP pipeline loads .pth + DataLoader → computes SHAP values → writes plots
c. Architectural Patterns & Principles
• Modular separation: each stage in its own module
• CLI + pipeline pattern: single-entrypoint dispatch to sub-pipelines
• Configurable hyperparameters via code injection (param_grid.py)
• Use of PyTorch’s Dataset/DataLoader abstraction
• Custom loss composition (CoxPH + KL) in VAE context
d. Key Technologies & Libraries
• Python 3.9+ for scripting
• PyTorch for model definition & training
• scikit-learn & scikit-survival for preprocessing & survival loss
• modified SHAP library for explainability
• AWS S3 (optional) for data storage