# hse-lambda/digital-twin architecture

> Architecture diagram of the GitHub repository hse-lambda/digital-twin, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hse-lambda/digital-twin
- Repository: https://github.com/hse-lambda/digital-twin

## Overview

1. **Purpose and scope**
   - This repository is a research/experiment codebase for building a “digital twin” of storage-system performance.
   - The core task is to predict two targets, **IOPS** and **latency**, from workload and hardware descriptors.

2. **Primary data boundary**
   - The raw source data is packaged in `dataset.tar.gz`.
   - The data model is tabular: features like `block_size`, `n_jobs`, `iodepth`, `read_fraction`, `load_type`, `io_type`, `raid`, `n_disks`, plus target columns `iops` and `lat`.
   - Train/test splits are generated and reused as CSVs; the repository also includes cached model artifacts and result archives, so the project is designed for offline reproduction rather than live serving.

3. **Library core: `digital_twin/data.py`**
   - This is the entry point for dataset handling and preprocessing.
   - It likely abstracts loading, splitting, and feature preparation so the experiment scripts and models consume a consistent tabular schema.

4. **Model family A: density estimation**
   - `digital_twin/models/density_estimation/` contains the classical generative-model baseline stack:
     - `gmm.py` for Gaussian mixture models,
     - `knn.py` for nearest-neighbor-based estimation,
     - `regressor.py` for supervised regression baselines,
     - `grouper.py` for grouping/conditioning logic.
   - Architecturally, this subsystem appears to map workload/device features to performance distributions or point predictions, then compare those predictions against observed measurements.

5. **Model family B: normalizing flows**
   - `digital_twin/models/norm_flow/` is the main generative modeling path in the paper.
   - `model.py` is the top-level model definition, while `utils/flow.py`, `transform.py`, and `metrics.py` support flow construction, variable transforms, and training/evaluation utilities.
   - `viz.py` suggests the flow subsystem also includes diagnostic plotting for learned distributions and prediction quality.

6. **Experiment orchestration**
   - The `scripts/` directory is the reproducibility layer.
   - `generate_train_test_split.py` prepares datasets; `calculate_summary_stats.py` aggregates metrics and prediction plots; `denesity_estimation_experiment.py`, `knn_experiment.py`, and `nf_experiment.py` run the main model families.
   - This means the workflow is: raw CSV data → split/preprocess → train model → write predictions/artifacts → summarize results.

7. **Evaluation and validation layer**
   - `digital_twin/performance_metrics/` contains the quantitative comparison tools:
     - `fd.py` for Fréchet distance,
     - `mmd.py` for maximum mean discrepancy,
     - `misc.py` for auxiliary metrics.
   - The README highlights Little’s law as an additional validation principle, so evaluation is not just point error but also distributional and physical-consistency checks.

8. **Visualization boundary**
   - `digital_twin/visulization/plots.py` provides plotting utilities for comparing predictions, distributions, and summary statistics.
   - The presence of notebooks and plot helpers suggests analysis is interactive and report-driven, not embedded in a runtime service.

9. **Persistent artifacts and checkpoints**
   - `models_checkpoints/` stores trained models and preprocessing artifacts:
     - CatBoost checkpoints,
     - KNN pickles,
     - normalizing-flow checkpoints plus scalers and categorical encoders.
   - The split between `cache/` and `pools/` indicates separate training artifacts for different dataset regimes such as cache vs storage pools and HDD/SSD workload types.

10. **Outputs and reproducibility**
   - `results/` contains archived experiment outputs, likely prediction CSVs, metrics, and plots bundled as tarballs.
   - `notebooks/` provides exploratory analysis and visualization, while `main.py` is likely a thin entry point or wrapper for running the project.
   - The overall architecture is offline, experiment-centric, and optimized for reproducible scientific comparison across model families.

11. **Operational/tooling layer**
   - `.devcontainer`, `Dockerfile`, `makefile`, `pyproject.toml`, and `.pre-commit-config.yaml` show the project is packaged for reproducible development and containerized execution.
   - This affects architecture mainly by standardizing the Python environment and making experiment replay stable across machines.

12. **How the pieces fit together**
   - Data flows from `dataset.tar.gz` into `data.py` and the split-generation script, then into the model-specific training scripts.
   - Trained models produce predictions and checkpoints, which are evaluated by the metrics package and summarized/visualized by the plotting and summary scripts.
   - The repository is therefore organized around a research pipeline: **dataset → preprocessing/splitting → model training → prediction/evaluation → plots/results**.

Main entry points: [`digital_twin/models/norm_flow/model.py`](https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/models/norm_flow/model.py) (Flow model), [`scripts/denesity_estimation_experiment.py`](https://github.com/hse-lambda/digital-twin/blob/master/scripts/denesity_estimation_experiment.py) (Density runs), [`digital_twin/data.py`](https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/data.py) (Data prep), [`scripts/generate_train_test_split.py`](https://github.com/hse-lambda/digital-twin/blob/master/scripts/generate_train_test_split.py) (Split data)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_data["Data boundary"]
  node_dataset[("Dataset<br/>raw data<br/>[dataset.tar.gz]")]
  node_data_py["Data prep<br/>preprocess<br/>[data.py]"]
end

subgraph group_group_models["Modeling"]
  node_gmm["GMM<br/>density baseline<br/>[gmm.py]"]
  node_knn["KNN<br/>density baseline<br/>[knn.py]"]
  node_regressor["Regressor<br/>supervised baseline<br/>[regressor.py]"]
  node_grouper["Grouper<br/>conditioning logic<br/>[grouper.py]"]
  node_nf_model["Flow model<br/>generative model<br/>[model.py]"]
  node_flow_utils["Flow utils<br/>flow construction<br/>[flow.py]"]
  node_transform["Transforms<br/>feature transform<br/>[transform.py]"]
  node_nf_metrics["Flow metrics<br/>training metrics<br/>[metrics.py]"]
  node_nf_viz["Flow viz<br/>diagnostics<br/>[viz.py]"]
end

subgraph group_group_eval["Evaluation"]
  node_fd["Fréchet<br/>metric<br/>[fd.py]"]
  node_mmd["MMD<br/>metric<br/>[mmd.py]"]
  node_plots["Plots<br/>visualization<br/>[plots.py]"]
end

subgraph group_group_runs["Experiments"]
  node_split_script["Split data<br/>experiment script"]
  node_de_script["Density runs<br/>experiment script"]
  node_knn_script["KNN run<br/>experiment script<br/>[knn_experiment.py]"]
  node_nf_script["Flow run<br/>experiment script<br/>[nf_experiment.py]"]
  node_summary_script["Summaries<br/>analysis script"]
  node_main["Main<br/>entry point<br/>[main.py]"]
end

subgraph group_group_artifacts["Artifacts"]
  node_checkpoints[("Checkpoints<br/>artifacts")]
  node_results[("Results<br/>archives")]
  node_notebooks["Notebooks"]
end

node_dataset -->|"load"| node_data_py
node_data_py -->|"split"| node_split_script
node_data_py -->|"features"| node_de_script
node_data_py -->|"features"| node_knn_script
node_data_py -->|"features"| node_nf_script
node_split_script -->|"train/test"| node_de_script
node_split_script -->|"train/test"| node_knn_script
node_split_script -->|"train/test"| node_nf_script
node_de_script -->|"fit"| node_gmm
node_de_script -->|"fit"| node_regressor
node_de_script -->|"condition"| node_grouper
node_knn_script -->|"fit"| node_knn
node_nf_script -->|"train"| node_nf_model
node_nf_model -->|"builds"| node_flow_utils
node_nf_model -->|"scales"| node_transform
node_nf_model -->|"evaluates"| node_nf_metrics
node_nf_model -->|"diagnose"| node_nf_viz
node_gmm -->|"score"| node_fd
node_knn -->|"score"| node_mmd
node_regressor -->|"plot"| node_plots
node_nf_model -->|"score"| node_fd
node_nf_model -->|"score"| node_mmd
node_fd -->|"aggregate"| node_summary_script
node_mmd -->|"aggregate"| node_summary_script
node_plots -->|"render"| node_summary_script
node_summary_script -->|"write"| node_results
node_de_script -->|"save"| node_checkpoints
node_knn_script -->|"save"| node_checkpoints
node_nf_script -->|"save"| node_checkpoints
node_notebooks -->|"inspect"| node_results
node_main -->|"orchestrate"| node_split_script

click node_dataset "https://github.com/hse-lambda/digital-twin/blob/master/dataset.tar.gz"
click node_data_py "https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/data.py"
click node_gmm "https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/models/density_estimation/gmm.py"
click node_knn "https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/models/density_estimation/knn.py"
click node_regressor "https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/models/density_estimation/regressor.py"
click node_grouper "https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/models/density_estimation/grouper.py"
click node_nf_model "https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/models/norm_flow/model.py"
click node_flow_utils "https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/models/norm_flow/utils/flow.py"
click node_transform "https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/models/norm_flow/utils/transform.py"
click node_nf_metrics "https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/models/norm_flow/utils/metrics.py"
click node_nf_viz "https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/models/norm_flow/utils/viz.py"
click node_fd "https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/performance_metrics/fd.py"
click node_mmd "https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/performance_metrics/mmd.py"
click node_plots "https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/visulization/plots.py"
click node_split_script "https://github.com/hse-lambda/digital-twin/blob/master/scripts/generate_train_test_split.py"
click node_de_script "https://github.com/hse-lambda/digital-twin/blob/master/scripts/denesity_estimation_experiment.py"
click node_knn_script "https://github.com/hse-lambda/digital-twin/blob/master/scripts/knn_experiment.py"
click node_nf_script "https://github.com/hse-lambda/digital-twin/blob/master/scripts/nf_experiment.py"
click node_summary_script "https://github.com/hse-lambda/digital-twin/blob/master/scripts/calculate_summary_stats.py"
click node_checkpoints "https://github.com/hse-lambda/digital-twin/tree/master/models_checkpoints"
click node_results "https://github.com/hse-lambda/digital-twin/tree/master/results"
click node_notebooks "https://github.com/hse-lambda/digital-twin/tree/master/notebooks"
click node_main "https://github.com/hse-lambda/digital-twin/blob/master/main.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dataset,node_data_py toneBlue
class node_gmm,node_knn,node_regressor,node_grouper,node_nf_model,node_flow_utils,node_transform,node_nf_metrics,node_nf_viz toneAmber
class node_fd,node_mmd,node_plots toneMint
class node_split_script,node_de_script,node_knn_script,node_nf_script,node_summary_script,node_main toneRose
class node_checkpoints,node_results,node_notebooks toneIndigo
```

## Components

### Data boundary

Raw dataset and preprocessing entry point

- [Dataset](https://github.com/hse-lambda/digital-twin/blob/master/dataset.tar.gz): raw data · Packaged tabular workload/performance data with feature and target columns · `dataset.tar.gz`
- [Data prep](https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/data.py): preprocess · Loads, splits, and standardizes the tabular schema for experiments · `digital_twin/data.py`

### Modeling

Baseline density estimators and normalizing-flow models

- [GMM](https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/models/density_estimation/gmm.py): density baseline · Gaussian mixture baseline for conditional performance estimation · `digital_twin/models/density_estimation/gmm.py`
- [KNN](https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/models/density_estimation/knn.py): density baseline · Nearest-neighbor estimator for performance prediction · `digital_twin/models/density_estimation/knn.py`
- [Regressor](https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/models/density_estimation/regressor.py): supervised baseline · Supervised regression baseline used alongside the density estimators · `digital_twin/models/density_estimation/regressor.py`
- [Grouper](https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/models/density_estimation/grouper.py): conditioning logic · Groups workloads or descriptors for conditional modeling · `digital_twin/models/density_estimation/grouper.py`
- [Flow model](https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/models/norm_flow/model.py): generative model · Top-level normalizing-flow model for IOPS and latency prediction · `digital_twin/models/norm_flow/model.py`
- [Flow utils](https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/models/norm_flow/utils/flow.py): flow construction · Builds flow components and training-time helpers · `digital_twin/models/norm_flow/utils/flow.py`
- [Transforms](https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/models/norm_flow/utils/transform.py): feature transform · Variable transforms and scaling used by the flow pipeline · `digital_twin/models/norm_flow/utils/transform.py`
- [Flow metrics](https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/models/norm_flow/utils/metrics.py): training metrics · Losses and evaluation helpers specific to the flow model · `digital_twin/models/norm_flow/utils/metrics.py`
- [Flow viz](https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/models/norm_flow/utils/viz.py): diagnostics · Visualization helpers for learned distributions and predictions · `digital_twin/models/norm_flow/utils/viz.py`

### Evaluation

Metrics, plots, and summary analysis

- [Fréchet](https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/performance_metrics/fd.py): metric · Distributional comparison metric · `digital_twin/performance_metrics/fd.py`
- [MMD](https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/performance_metrics/mmd.py): metric · Maximum mean discrepancy for comparing predicted and observed distributions · `digital_twin/performance_metrics/mmd.py`
- [Plots](https://github.com/hse-lambda/digital-twin/blob/master/digital_twin/visulization/plots.py): visualization · Shared plotting utilities for predictions and summary statistics · `digital_twin/visulization/plots.py`

### Experiments

Scripts that generate splits, train models, and summarize results

- [Split data](https://github.com/hse-lambda/digital-twin/blob/master/scripts/generate_train_test_split.py): experiment script · Generates and persists train/test splits · `scripts/generate_train_test_split.py`
- [Density runs](https://github.com/hse-lambda/digital-twin/blob/master/scripts/denesity_estimation_experiment.py): experiment script · Runs density-estimation and regression baselines · `scripts/denesity_estimation_experiment.py`
- [KNN run](https://github.com/hse-lambda/digital-twin/blob/master/scripts/knn_experiment.py): experiment script · Runs KNN baseline experiments · `scripts/knn_experiment.py`
- [Flow run](https://github.com/hse-lambda/digital-twin/blob/master/scripts/nf_experiment.py): experiment script · Trains and evaluates the normalizing-flow experiments · `scripts/nf_experiment.py`
- [Summaries](https://github.com/hse-lambda/digital-twin/blob/master/scripts/calculate_summary_stats.py): analysis script · Aggregates metrics and prediction plots across runs · `scripts/calculate_summary_stats.py`
- [Main](https://github.com/hse-lambda/digital-twin/blob/master/main.py): entry point · Thin project entry point or wrapper · `main.py`

### Artifacts

Cached checkpoints, notebooks, and archived outputs

- [Checkpoints](https://github.com/hse-lambda/digital-twin/tree/master/models_checkpoints): artifacts · Saved model weights, encoders, and scalers for offline reuse · `models_checkpoints`
- [Results](https://github.com/hse-lambda/digital-twin/tree/master/results): archives · Archived predictions, metrics, and experiment outputs · `results`
- [Notebooks](https://github.com/hse-lambda/digital-twin/tree/master/notebooks): analysis notebooks · Interactive exploration, metrics inspection, and visualization notebooks · `notebooks`

## Connections

- Dataset → Data prep: load · Raw data is loaded and normalized into the project schema
- Data prep → Split data: split · Preprocessed data is split into reusable train/test sets
- Data prep → Density runs: features · Experiment scripts consume the shared tabular representation
- Data prep → KNN run: features · Experiment scripts consume the shared tabular representation
- Data prep → Flow run: features · Experiment scripts consume the shared tabular representation
- Split data → Density runs: train/test · Generated splits feed baseline training runs
- Split data → KNN run: train/test · Generated splits feed KNN experiments
- Split data → Flow run: train/test · Generated splits feed flow experiments
- Density runs → GMM: fit · Baseline experiments train the generative estimators
- Density runs → Regressor: fit · Baseline experiments train supervised regressors
- Density runs → Grouper: condition · Grouping logic supports conditional baselines
- KNN run → KNN: fit · KNN experiments train nearest-neighbor estimators
- Flow run → Flow model: train · Flow experiments train the main generative model
- Flow model → Flow utils: builds · The model is assembled from flow construction helpers
- Flow model → Transforms: scales · Feature and target transforms support training and inference
- Flow model → Flow metrics: evaluates · Model training relies on flow-specific metrics and losses
- Flow model → Flow viz: diagnose · Diagnostics inspect learned distributions and predictions
- GMM → Fréchet: score · Distributional outputs are compared with Fréchet distance
- KNN → MMD: score · Predictions can be compared with MMD-style metrics
- Regressor → Plots: plot · Regression outputs are visualized for analysis
- Flow model → Fréchet: score · Flow outputs are validated with distributional metrics
- Flow model → MMD: score · Flow outputs are validated with distributional metrics
- Fréchet → Summaries: aggregate · Metrics are summarized across experiment runs
- MMD → Summaries: aggregate · Metrics are summarized across experiment runs
- Plots → Summaries: render · Plots are bundled into result summaries
- Summaries → Results: write · Summaries and plots are archived as experiment outputs
- Density runs → Checkpoints: save · Baseline model artifacts are persisted for reuse
- KNN run → Checkpoints: save · KNN artifacts are persisted for reuse
- Flow run → Checkpoints: save · Flow checkpoints, scalers, and encoders are persisted
- Notebooks → Results: inspect · Notebooks consume archived outputs for exploration
- Main → Split data: orchestrate · Main entry point likely dispatches the experiment workflow
