How hse-lambda/digital-twin works
1. Purpose and scope
- This repository is a research/experiment codebase for building a “digital twin” of storage-system performance.
- The core task is to predict two targets, IOPS and latency, from workload and hardware descriptors.
2. Primary data boundary
- The raw source data is packaged in dataset.tar.gz.
- The data model is tabular: features like block_size, n_jobs, iodepth, read_fraction, load_type, io_type, raid, n_disks, plus target columns iops and lat.
- Train/test splits are generated and reused as CSVs; the repository also includes cached model artifacts and result archives, so the project is designed for offline reproduction rather than live serving.
3. Library core: digital_twin/data.py
- This is the entry point for dataset handling and preprocessing.
- It likely abstracts loading, splitting, and feature preparation so the experiment scripts and models consume a consistent tabular schema.
4. Model family A: density estimation
- digital_twin/models/density_estimation/ contains the classical generative-model baseline stack:
- gmm.py for Gaussian mixture models,
- knn.py for nearest-neighbor-based estimation,
- regressor.py for supervised regression baselines,
- grouper.py for grouping/conditioning logic.
- Architecturally, this subsystem appears to map workload/device features to performance distributions or point predictions, then compare those predictions against observed measurements.
5. Model family B: normalizing flows
- digital_twin/models/norm_flow/ is the main generative modeling path in the paper.
- model.py is the top-level model definition, while utils/flow.py, transform.py, and metrics.py support flow construction, variable transforms, and training/evaluation utilities.
- viz.py suggests the flow subsystem also includes diagnostic plotting for learned distributions and prediction quality.
6. Experiment orchestration
- The scripts/ directory is the reproducibility layer.
- generate_train_test_split.py prepares datasets; calculate_summary_stats.py aggregates metrics and prediction plots; denesity_estimation_experiment.py, knn_experiment.py, and nf_experiment.py run the main model families.
- This means the workflow is: raw CSV data → split/preprocess → train model → write predictions/artifacts → summarize results.
7. Evaluation and validation layer
- digital_twin/performance_metrics/ contains the quantitative comparison tools:
- fd.py for Fréchet distance,
- mmd.py for maximum mean discrepancy,
- misc.py for auxiliary metrics.
- The README highlights Little’s law as an additional validation principle, so evaluation is not just point error but also distributional and physical-consistency checks.
8. Visualization boundary
- digital_twin/visulization/plots.py provides plotting utilities for comparing predictions, distributions, and summary statistics.
- The presence of notebooks and plot helpers suggests analysis is interactive and report-driven, not embedded in a runtime service.
9. Persistent artifacts and checkpoints
- models_checkpoints/ stores trained models and preprocessing artifacts:
- CatBoost checkpoints,
- KNN pickles,
- normalizing-flow checkpoints plus scalers and categorical encoders.
- The split between cache/ and pools/ indicates separate training artifacts for different dataset regimes such as cache vs storage pools and HDD/SSD workload types.
10. Outputs and reproducibility
- results/ contains archived experiment outputs, likely prediction CSVs, metrics, and plots bundled as tarballs.
- notebooks/ provides exploratory analysis and visualization, while main.py is likely a thin entry point or wrapper for running the project.
- The overall architecture is offline, experiment-centric, and optimized for reproducible scientific comparison across model families.
11. Operational/tooling layer
- .devcontainer, Dockerfile, makefile, pyproject.toml, and .pre-commit-config.yaml show the project is packaged for reproducible development and containerized execution.
- This affects architecture mainly by standardizing the Python environment and making experiment replay stable across machines.
12. How the pieces fit together
- Data flows from dataset.tar.gz into data.py and the split-generation script, then into the model-specific training scripts.
- Trained models produce predictions and checkpoints, which are evaluated by the metrics package and summarized/visualized by the plotting and summary scripts.
- The repository is therefore organized around a research pipeline: dataset → preprocessing/splitting → model training → prediction/evaluation → plots/results.
Main entry points: model.py Flow model · denesity_estimation_experiment.py Density runs · data.py Data prep · generate_train_test_split.py Split data
Components 23
Data boundary
Raw dataset and preprocessing entry point
Modeling
Baseline density estimators and normalizing-flow models
- GMM · Gaussian mixture baseline for conditional performance estimation
digital_twin/models/density_estimation/gmm.py - KNN · Nearest-neighbor estimator for performance prediction
digital_twin/models/density_estimation/knn.py - Regressor · Supervised regression baseline used alongside the density estimators
digital_twin/models/density_estimation/regressor.py - Grouper · Groups workloads or descriptors for conditional modeling
digital_twin/models/density_estimation/grouper.py - Flow model · Top-level normalizing-flow model for IOPS and latency prediction
digital_twin/models/norm_flow/model.py - Flow utils · Builds flow components and training-time helpers
digital_twin/models/norm_flow/utils/flow.py - Transforms · Variable transforms and scaling used by the flow pipeline
digital_twin/models/norm_flow/utils/transform.py - Flow metrics · Losses and evaluation helpers specific to the flow model
digital_twin/models/norm_flow/utils/metrics.py - Flow viz · Visualization helpers for learned distributions and predictions
digital_twin/models/norm_flow/utils/viz.py
Evaluation
Metrics, plots, and summary analysis
- Fréchet · Distributional comparison metric
digital_twin/performance_metrics/fd.py - MMD · Maximum mean discrepancy for comparing predicted and observed distributions
digital_twin/performance_metrics/mmd.py - Plots · Shared plotting utilities for predictions and summary statistics
digital_twin/visulization/plots.py
Experiments
Scripts that generate splits, train models, and summarize results
- Split data · Generates and persists train/test splits
scripts/generate_train_test_split.py - Density runs · Runs density-estimation and regression baselines
scripts/denesity_estimation_experiment.py - KNN run · Runs KNN baseline experiments
scripts/knn_experiment.py - Flow run · Trains and evaluates the normalizing-flow experiments
scripts/nf_experiment.py - Summaries · Aggregates metrics and prediction plots across runs
scripts/calculate_summary_stats.py - Main · Thin project entry point or wrapper
main.py
Artifacts
Cached checkpoints, notebooks, and archived outputs
- Checkpoints · Saved model weights, encoders, and scalers for offline reuse
models_checkpoints - Results · Archived predictions, metrics, and experiment outputs
results - Notebooks · Interactive exploration, metrics inspection, and visualization notebooks
notebooks
Connections 31
- Dataset → Data prep: load · Raw data is loaded and normalized into the project schema
- Data prep → Split data: split · Preprocessed data is split into reusable train/test sets
- Data prep → Density runs: features · Experiment scripts consume the shared tabular representation
- Data prep → KNN run: features · Experiment scripts consume the shared tabular representation
- Data prep → Flow run: features · Experiment scripts consume the shared tabular representation
- Split data → Density runs: train/test · Generated splits feed baseline training runs
- Split data → KNN run: train/test · Generated splits feed KNN experiments
- Split data → Flow run: train/test · Generated splits feed flow experiments
- Density runs → GMM: fit · Baseline experiments train the generative estimators
- Density runs → Regressor: fit · Baseline experiments train supervised regressors
- Density runs → Grouper: condition · Grouping logic supports conditional baselines
- KNN run → KNN: fit · KNN experiments train nearest-neighbor estimators
- Flow run → Flow model: train · Flow experiments train the main generative model
- Flow model → Flow utils: builds · The model is assembled from flow construction helpers
- Flow model → Transforms: scales · Feature and target transforms support training and inference
- Flow model → Flow metrics: evaluates · Model training relies on flow-specific metrics and losses
- Flow model → Flow viz: diagnose · Diagnostics inspect learned distributions and predictions
- GMM → Fréchet: score · Distributional outputs are compared with Fréchet distance
- KNN → MMD: score · Predictions can be compared with MMD-style metrics
- Regressor → Plots: plot · Regression outputs are visualized for analysis
- Flow model → Fréchet: score · Flow outputs are validated with distributional metrics
- Flow model → MMD: score · Flow outputs are validated with distributional metrics
- Fréchet → Summaries: aggregate · Metrics are summarized across experiment runs
- MMD → Summaries: aggregate · Metrics are summarized across experiment runs
- Plots → Summaries: render · Plots are bundled into result summaries
- Summaries → Results: write · Summaries and plots are archived as experiment outputs
- Density runs → Checkpoints: save · Baseline model artifacts are persisted for reuse
- KNN run → Checkpoints: save · KNN artifacts are persisted for reuse
- Flow run → Checkpoints: save · Flow checkpoints, scalers, and encoders are persisted
- Notebooks → Results: inspect · Notebooks consume archived outputs for exploration
- Main → Split data: orchestrate · Main entry point likely dispatches the experiment workflow