# jiagengchang/myevae architecture

> Architecture diagram of the GitHub repository jiagengchang/myevae, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jiagengchang/myevae
- Repository: https://github.com/jiagengchang/myevae

## Overview

##### Project Type and Purpose
- MyeVAE is a Python-based CLI tool and library implementing a multi-modal Variational Autoencoder for risk prediction in multiple myeloma patients.
- Core workflows: feature preprocessing, model training, scoring, and SHAP analysis—each invoked via `myevae` commands.

##### File-Structure Analysis
- Top-level:
- LICENSE, README.md, pyproject.toml, requirements.txt (project metadata & dependencies)
- assets/ (illustrations)
- src/myevae/ – code modules
- Key modules under src/myevae:
- `__main__.py` — CLI entrypoint parses subcommands (`preprocess`, `train`, `score`, `shap`)
- `pipelinetools.py` — orchestrates end-to-end pipelines per subcommand
- `preprocess.py` — feature selection (elastic net CoxPH), imputation (IterativeImputer)
- `dataset.py` — PyTorch Dataset/DataLoader wrappers for CSV inputs
- `buildnetwork.py` & `nn.py` — constructs the VAE encoder-decoder graph, including multi-omics subtasks
- `coxphloss.py`, `kldivergence.py` — custom loss components (Cox proportional hazards & KL divergence)
- `train.py` & `estimator.py` — training loops, early stopping, hyperparameter search via `param_grid.py`
- `score.py` — loads trained model, computes validation/test metrics
- `shap.py` & `plotshap.py` — invokes SHAP value computation and plotting
- `param_grid.py` — default hyperparameter grid for testing; override via CLI.

##### README Insights
- Four sequential CLI stages: preprocess → train → score → shap
- Inputs: raw or preprocessed CSVs + optional custom `param_grid.py`
- Outputs: processed CSVs, model weights (`.pth`), metrics (`.json`), SHAP plots
- Dependencies: Python ≥ 3.9, PyTorch, scikit-learn, scikit-survival, modified SHAP.

a. Main Components
1. CLI Layer (`__main__.py`)
2. Pipeline Orchestrator (`pipelinetools.py`)
3. Data Layer
- Raw CSV inputs
- `preprocess.py` → processed CSVs
- `dataset.py` → DataLoader
4. Model Layer
- `buildnetwork.py` + `nn.py` (VAE architecture)
- Loss modules (`coxphloss.py`, `kldivergence.py`)
5. Training & Evaluation
- `estimator.py` & `train.py` (fit, grid search, early stopping)
- `score.py` (metrics computation)
6. Explainability
- `shap.py` & `plotshap.py` (SHAP value generation, visualizations)
7. Configuration
- `param_grid.py` (hyperparams)
8. External Data Store (optional)
- AWS S3 buckets for example inputs/outputs

b. Relationships & Interactions
- CLI → Pipeline Orchestrator
- Orchestrator invokes either preprocessing, training, scoring or SHAP pipelines
- Preprocessing pipeline reads raw CSVs → writes processed CSVs
- Training pipeline reads processed CSVs + `param_grid.py` → builds network → trains model → writes `.pth` weights + metrics
- Scoring pipeline loads `.pth` → computes metrics → writes `.json`
- SHAP pipeline loads `.pth` + DataLoader → computes SHAP values → writes plots

c. Architectural Patterns & Principles
- Modular separation: each stage in its own module
- CLI + pipeline pattern: single-entrypoint dispatch to sub-pipelines
- Configurable hyperparameters via code injection (`param_grid.py`)
- Use of PyTorch’s Dataset/DataLoader abstraction
- Custom loss composition (CoxPH + KL) in VAE context

d. Key Technologies & Libraries
- Python 3.9+ for scripting
- PyTorch for model definition & training
- scikit-learn & scikit-survival for preprocessing & survival loss
- modified SHAP library for explainability
- AWS S3 (optional) for data storage

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI & Orchestrator Layer
    CLI["CLI Layer (__main__.py)"]:::cli
    Orchestrator["Pipeline Orchestrator (pipelinetools.py)"]:::cli

    %% Data Layer
    RawCSV["Raw CSVs"]:::data
    Preprocess["Preprocessing Module (preprocess.py)"]:::data
    RefData["Reference Data (ensembl_ref.csv)"]:::data
    ProcessedCSV["Processed CSVs"]:::data
    Dataset["Dataset/DataLoader (dataset.py)"]:::data

    %% Model Layer
    BuildNet["VAE Architecture (buildnetwork.py)"]:::model
    NN["Module Definitions (nn.py)"]:::model
    Loss1["CoxPH Loss (coxphloss.py)"]:::model
    Loss2["KL Divergence Loss (kldivergence.py)"]:::model

    %% Training & Evaluation Layer
    Train["Training Orchestration (train.py)"]:::eval
    Estimator["Estimator (estimator.py)"]:::eval
    ParamGrid["Hyperparameter Grid (param_grid.py)"]:::config
    ModelWeights["Model Weights (.pth)"]:::data
    TrainMetrics["Training Metrics (.json)"]:::data

    %% Scoring Module
    Score["Scoring Module (score.py)"]:::eval
    ScoreMetrics["Score Metrics (.json)"]:::data

    %% Explainability Layer
    SHAPComp["SHAP Computation (shap.py)"]:::eval
    PlotShap["SHAP Plotting (plotshap.py)"]:::eval
    SHAPValues["SHAP Values"]:::data
    SHAPPlots["SHAP Plots"]:::data

    %% External Storage
    S3["AWS S3"]:::external

    %% Connections
    CLI -->|invokes| Orchestrator

    Orchestrator -->|start preprocess| Preprocess
    RawCSV -->|reads| Preprocess
    RefData -->|reads| Preprocess
    Preprocess -->|writes| ProcessedCSV

    Orchestrator -->|start train| Train
    ProcessedCSV -->|reads| Train
    ParamGrid -->|reads| Train
    BuildNet -->|defines network| Train
    Loss1 -->|used by| Train
    Loss2 -->|used by| Train
    Train -->|calls| Estimator
    Estimator -->|outputs| ModelWeights
    Estimator -->|outputs| TrainMetrics

    Orchestrator -->|start score| Score
    ModelWeights -->|loads| Score
    ProcessedCSV -->|reads| Score
    Score -->|writes| ScoreMetrics

    Orchestrator -->|start shap| SHAPComp
    ModelWeights -->|loads| SHAPComp
    Dataset -->|provides data| SHAPComp
    SHAPComp -->|outputs| SHAPValues
    SHAPValues -->|inputs| PlotShap
    PlotShap -->|writes| SHAPPlots

    %% Optional S3 interactions (dashed)
    RawCSV -.->|sync| S3
    ProcessedCSV -.->|sync| S3
    ModelWeights -.->|sync| S3
    TrainMetrics -.->|sync| S3
    ScoreMetrics -.->|sync| S3
    SHAPPlots -.->|sync| S3

    %% Click Events
    click CLI "https://github.com/jiagengchang/myevae/blob/main/src/myevae/__main__.py"
    click Orchestrator "https://github.com/jiagengchang/myevae/blob/main/src/myevae/pipelinetools.py"
    click Preprocess "https://github.com/jiagengchang/myevae/blob/main/src/myevae/preprocess.py"
    click RefData "https://github.com/jiagengchang/myevae/blob/main/src/myevae/data/ensembl_ref.csv"
    click Dataset "https://github.com/jiagengchang/myevae/blob/main/src/myevae/dataset.py"
    click BuildNet "https://github.com/jiagengchang/myevae/blob/main/src/myevae/buildnetwork.py"
    click NN "https://github.com/jiagengchang/myevae/blob/main/src/myevae/nn.py"
    click Loss1 "https://github.com/jiagengchang/myevae/blob/main/src/myevae/coxphloss.py"
    click Loss2 "https://github.com/jiagengchang/myevae/blob/main/src/myevae/kldivergence.py"
    click Train "https://github.com/jiagengchang/myevae/blob/main/src/myevae/train.py"
    click Estimator "https://github.com/jiagengchang/myevae/blob/main/src/myevae/estimator.py"
    click ParamGrid "https://github.com/jiagengchang/myevae/blob/main/src/myevae/param_grid.py"
    click Score "https://github.com/jiagengchang/myevae/blob/main/src/myevae/score.py"
    click SHAPComp "https://github.com/jiagengchang/myevae/blob/main/src/myevae/shap.py"
    click PlotShap "https://github.com/jiagengchang/myevae/blob/main/src/myevae/plotshap.py"
    click S3 "https://github.com/jiagengchang/myevae/tree/main/AWS S3"

    %% Styles
    classDef cli fill:#08519c,stroke:#0492c2,color:#fff;
    classDef data fill:#238b45,stroke:#2ca02c,color:#fff;
    classDef model fill:#d6604d,stroke:#e6550d,color:#fff;
    classDef eval fill:#6a51a3,stroke:#807dba,color:#fff;
    classDef config fill:#17becf,stroke:#1f77b4,color:#fff;
    classDef external fill:#7f7f7f,stroke:#636363,color:#fff;
```
