# pauldasheinrich/idp architecture

> Architecture diagram of the GitHub repository pauldasheinrich/idp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pauldasheinrich/idp
- Repository: https://github.com/pauldasheinrich/idp

## Overview

1. **Overall shape**
   This repository is an end-to-end sleep modeling pipeline, not just a model package. It turns raw wearable/device logs into labeled nightly records, engineers predictors, generates synthetic augmentation data, trains/evaluates several regressors, and stores both final models and analysis outputs.

2. **Raw data ingress**
   The starting point is `data/raw/`, which contains device log text files for a small set of participants. The `src.data.raw_logs` module is the first pipeline stage and is responsible for parsing those logs into a structured intermediate form.

3. **Core preprocessing and labeling**
   The main real-data processing chain is:
   `raw_logs -> preprocess -> wake_state -> sleep_labels -> features`
   These stages live under `src/data/` and progressively clean the logs, derive wake/sleep state, create sleep labels, and build nightly feature tables. The repository’s permanent real-data feature inputs are `data/features/features_real.csv` and `data/windows/evening_windows.csv`.

4. **Feature boundary and leakage control**
   Feature engineering is deliberately constrained to pre-bedtime windows only, using `bedtime_proxy - 4 hours` to `bedtime_proxy`. Light is explicitly allowed only as a predictor, not as an input to wake/sleep label derivation. This is an important architectural boundary because the pipeline is designed to avoid label leakage.

5. **Synthetic data augmentation subsystem**
   `src/augmentation/generate.py` produces synthetic training data under `data/augmented/`. There are two augmentation modes:
   - `real_plus_synthetic_nights`: moving-block bootstrap over 10-minute evening windows
   - `real_plus_synthetic_participants`: pseudo-participants with coherent synthetic nights, participant-level perturbations, temporal drift, and conservative SOL shifts
   Synthetic artifacts are kept separate from real inputs, with outputs like `synthetic_features.csv`, `synthetic_evening_windows.csv`, and `synthetic_participant_profiles.csv`.

6. **Model-ready dataset assembly**
   `src.data.model_data` combines the engineered real data and the chosen synthetic augmentation into training/evaluation tables. The repository shows both the final merged dataset in `data/model/` and fold-specific splits in `data/results/folds/`. This is where the pipeline materializes training conditions such as real-only vs. real-plus-synthetic variants.

7. **Evaluation design and split strategies**
   Evaluation is fold-based and supports participant and temporal holdouts. The `data/results/folds/` directory contains fold-specific train/test CSVs for multiple scenarios, including:
   - participant holdout
   - temporal holdout
   - per-participant folds
   - global temporal holdout
   For each fold, the pipeline creates separate training sets for real-only, real-plus-synthetic-nights, and real-plus-synthetic-participants conditions. Held-out test participants are never augmented.

8. **Modeling subsystem**
   The modeling code lives in `src/models/` and supports three model families:
   - GAM
   - Random Forest
   - XGBoost
   These are trained and evaluated under configurable feature sets, target transforms, and hyperparameter grids defined in `config.json` and consumed by `src.models.config` and related modules.

9. **Experiment outputs and artifacts**
   Evaluation writes predictions, tuning results, summary tables, and rankings to `data/results/`. Important artifacts include:
   - `augmentation_model_results.csv`
   - `augmentation_fold_predictions.csv`
   - `augmentation_summary.csv`
   - `tuning_trials.csv` / `tuning_best_params.csv`
   - `augmentation_tuning_trials.csv` / `augmentation_tuning_best.csv`
   - `model_rmse_ranking.csv`
   Model artifacts are persisted under `models/` as pickled estimators, with separate files for real-only and augmented training regimes.

10. **Pipeline orchestration**
   `run_pipeline.py` is the top-level orchestrator. It performs a two-pass workflow: first it searches augmentation/model settings and evaluates them, then it selects the best augmentation configuration, regenerates augmented data, reruns final evaluation, and trains the final model payloads. The `--skip-finalize` flag stops after the initial search/evaluation pass.

11. **User-facing analysis and reporting**
   The repository includes a lightweight reporting layer in `reports/` and plotting utilities such as `plot_model_rmse.py`. Final report content is stored as Markdown and figures summarize model performance metrics like RMSE, MAE, R², and error statistics.

12. **Frontend boundary**
   `frontend/streamlit_app.py` is a separate Streamlit-based UI that reads pipeline outputs rather than recomputing the pipeline itself. It has its own `requirements.txt` and `pipeline_defaults.json`, so it functions as a consumer of the modeling outputs and default configuration, not as the core processing engine.

13. **Development and provenance**
   The `notebooks/` directory captures the same pipeline stages in exploratory form, mirroring the production modules. These notebooks document the analysis path but the production architecture is the Python module pipeline plus the orchestrating runner.

14. **Key architectural constraint**
   The most important system boundary is between real data and synthetic data: synthetic rows are used only for training, never for held-out evaluation. Final evaluation is always on real participant data from `data/model/model_table_real.csv`, which keeps the performance estimates anchored to observed data.

Main entry points: [`src/data/model_data.py`](https://github.com/pauldasheinrich/idp/blob/master/src/data/model_data.py) (Model data), [`src/models/train.py`](https://github.com/pauldasheinrich/idp/blob/master/src/models/train.py) (Train), [`data/results`](https://github.com/pauldasheinrich/idp/tree/master/data/results) (Results), [`run_pipeline.py`](https://github.com/pauldasheinrich/idp/blob/master/run_pipeline.py) (Pipeline runner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_ingest["Data prep"]
  node_n_raw_logs["Raw logs<br/>ingest parser<br/>[raw_logs.py]"]
  node_n_preprocess["Preprocess<br/>cleaning stage<br/>[preprocess.py]"]
  node_n_wake_state["Wake state<br/>state derivation<br/>[wake_state.py]"]
  node_n_sleep_labels["Sleep labels<br/>labeling stage<br/>[sleep_labels.py]"]
  node_n_features["Night features<br/>feature builder<br/>[features.py]"]
  node_n_windows["Evening windows<br/>windowing<br/>[windows.py]"]
end

subgraph group_group_aug["Augmentation"]
  node_n_augment["Synthetic gen<br/>augmentation pipeline<br/>[generate.py]"]
  node_n_aug_nights["Night bootstrap<br/>augmentation mode<br/>[generate.py]"]
  node_n_aug_parts["Participant synth<br/>augmentation mode<br/>[generate.py]"]
end

subgraph group_group_model["Modeling"]
  node_n_model_data["Model data<br/>dataset assembly<br/>[model_data.py]"]
  node_n_train["Train<br/>[train.py]"]
  node_n_evaluate["Evaluate<br/>evaluation<br/>[evaluate.py]"]
  node_n_gam["GAM<br/>model family<br/>[gam.py]"]
  node_n_rf["Random Forest<br/>model family<br/>[random_forest.py]"]
  node_n_xgb["XGBoost<br/>model family<br/>[xgboost.py]"]
end

subgraph group_group_output["Outputs"]
  node_n_results[("Results<br/>artifacts")]
  node_n_models[("Model files<br/>artifacts")]
  node_n_report["Report<br/>[final_report.md]"]
  node_n_frontend["Streamlit app<br/>consumer UI<br/>[streamlit_app.py]"]
end

subgraph group_group_orch["Orchestration"]
  node_n_config["Config<br/>settings<br/>[config.json]"]
  node_n_runner["Pipeline runner<br/>orchestrator<br/>[run_pipeline.py]"]
end

node_n_raw_logs -->|"parse to clean"| node_n_preprocess
node_n_preprocess -->|"derive state"| node_n_wake_state
node_n_wake_state -->|"label nights"| node_n_sleep_labels
node_n_sleep_labels -->|"bound windows"| node_n_windows
node_n_windows -->|"engineer features"| node_n_features
node_n_features -->|"real features"| node_n_model_data
node_n_augment -->|"mode"| node_n_aug_nights
node_n_augment -->|"mode"| node_n_aug_parts
node_n_aug_nights -->|"augment train"| node_n_model_data
node_n_aug_parts -->|"augment train"| node_n_model_data
node_n_config -->|"control"| node_n_runner
node_n_runner -->|"finalize augmentation"| node_n_augment
node_n_runner -->|"assemble sets"| node_n_model_data
node_n_model_data -->|"train data"| node_n_train
node_n_model_data -->|"eval data"| node_n_evaluate
node_n_train -->|"fit"| node_n_gam
node_n_train -->|"fit"| node_n_rf
node_n_train -->|"fit"| node_n_xgb
node_n_evaluate -->|"write metrics"| node_n_results
node_n_train -->|"persist"| node_n_models
node_n_results -->|"summarize"| node_n_report
node_n_results -->|"consume"| node_n_frontend
node_n_models -.->|"load models"| node_n_frontend

click node_n_raw_logs "https://github.com/pauldasheinrich/idp/blob/master/src/data/raw_logs.py"
click node_n_preprocess "https://github.com/pauldasheinrich/idp/blob/master/src/data/preprocess.py"
click node_n_wake_state "https://github.com/pauldasheinrich/idp/blob/master/src/data/wake_state.py"
click node_n_sleep_labels "https://github.com/pauldasheinrich/idp/blob/master/src/data/sleep_labels.py"
click node_n_features "https://github.com/pauldasheinrich/idp/blob/master/src/data/features.py"
click node_n_windows "https://github.com/pauldasheinrich/idp/blob/master/src/data/windows.py"
click node_n_model_data "https://github.com/pauldasheinrich/idp/blob/master/src/data/model_data.py"
click node_n_augment "https://github.com/pauldasheinrich/idp/blob/master/src/augmentation/generate.py"
click node_n_aug_nights "https://github.com/pauldasheinrich/idp/blob/master/src/augmentation/generate.py"
click node_n_aug_parts "https://github.com/pauldasheinrich/idp/blob/master/src/augmentation/generate.py"
click node_n_config "https://github.com/pauldasheinrich/idp/blob/master/config.json"
click node_n_runner "https://github.com/pauldasheinrich/idp/blob/master/run_pipeline.py"
click node_n_train "https://github.com/pauldasheinrich/idp/blob/master/src/models/train.py"
click node_n_evaluate "https://github.com/pauldasheinrich/idp/blob/master/src/models/evaluate.py"
click node_n_gam "https://github.com/pauldasheinrich/idp/blob/master/src/models/gam.py"
click node_n_rf "https://github.com/pauldasheinrich/idp/blob/master/src/models/random_forest.py"
click node_n_xgb "https://github.com/pauldasheinrich/idp/blob/master/src/models/xgboost.py"
click node_n_results "https://github.com/pauldasheinrich/idp/tree/master/data/results"
click node_n_models "https://github.com/pauldasheinrich/idp/tree/master/models"
click node_n_report "https://github.com/pauldasheinrich/idp/blob/master/reports/final_report.md"
click node_n_frontend "https://github.com/pauldasheinrich/idp/blob/master/frontend/streamlit_app.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_raw_logs,node_n_preprocess,node_n_wake_state,node_n_sleep_labels,node_n_features,node_n_windows toneBlue
class node_n_augment,node_n_aug_nights,node_n_aug_parts toneAmber
class node_n_model_data,node_n_train,node_n_evaluate,node_n_gam,node_n_rf,node_n_xgb toneMint
class node_n_results,node_n_models,node_n_report,node_n_frontend toneRose
class node_n_config,node_n_runner toneIndigo
```

## Components

### Data prep

Raw logs, preprocessing, labels, and feature assembly for nightly sleep records.

- [Raw logs](https://github.com/pauldasheinrich/idp/blob/master/src/data/raw_logs.py): ingest parser · Parses wearable/device text logs into a structured intermediate form. · `src/data/raw_logs.py`
- [Preprocess](https://github.com/pauldasheinrich/idp/blob/master/src/data/preprocess.py): cleaning stage · Cleans and standardizes the parsed logs into processed records. · `src/data/preprocess.py`
- [Wake state](https://github.com/pauldasheinrich/idp/blob/master/src/data/wake_state.py): state derivation · Derives wake/sleep state used by downstream labeling. · `src/data/wake_state.py`
- [Sleep labels](https://github.com/pauldasheinrich/idp/blob/master/src/data/sleep_labels.py): labeling stage · Builds nightly sleep labels from processed state information. · `src/data/sleep_labels.py`
- [Night features](https://github.com/pauldasheinrich/idp/blob/master/src/data/features.py): feature builder · Creates pre-bedtime nightly feature tables with leakage-aware windows. · `src/data/features.py`
- [Evening windows](https://github.com/pauldasheinrich/idp/blob/master/src/data/windows.py): windowing · Defines pre-bedtime evening windows used for feature construction. · `src/data/windows.py`

### Augmentation

Synthetic data generation and augmentation modes used for training only.

- [Synthetic gen](https://github.com/pauldasheinrich/idp/blob/master/src/augmentation/generate.py): augmentation pipeline · Generates synthetic training data in nightly and participant-level modes. · `src/augmentation/generate.py`
- [Night bootstrap](https://github.com/pauldasheinrich/idp/blob/master/src/augmentation/generate.py): augmentation mode · Creates synthetic nights with moving-block bootstrap over evening windows. · `src/augmentation/generate.py`
- [Participant synth](https://github.com/pauldasheinrich/idp/blob/master/src/augmentation/generate.py): augmentation mode · Creates pseudo-participants with coherent synthetic nights and drift. · `src/augmentation/generate.py`

### Modeling

Dataset assembly, training, tuning, and evaluation of regressors.

- [Model data](https://github.com/pauldasheinrich/idp/blob/master/src/data/model_data.py): dataset assembly · Merges real features with selected augmentation into model-ready tables. · `src/data/model_data.py`
- [Train](https://github.com/pauldasheinrich/idp/blob/master/src/models/train.py): training · Fits GAM, Random Forest, and XGBoost models under selected conditions. · `src/models/train.py`
- [Evaluate](https://github.com/pauldasheinrich/idp/blob/master/src/models/evaluate.py): evaluation · Runs fold-based evaluation, tuning, and summary ranking. · `src/models/evaluate.py`
- [GAM](https://github.com/pauldasheinrich/idp/blob/master/src/models/gam.py): model family · Generalized additive model implementation and training logic. · `src/models/gam.py`
- [Random Forest](https://github.com/pauldasheinrich/idp/blob/master/src/models/random_forest.py): model family · Random forest regression implementation and training logic. · `src/models/random_forest.py`
- [XGBoost](https://github.com/pauldasheinrich/idp/blob/master/src/models/xgboost.py): model family · XGBoost regression implementation and training logic. · `src/models/xgboost.py`

### Outputs

Persisted results, models, reports, and the Streamlit consumer.

- [Results](https://github.com/pauldasheinrich/idp/tree/master/data/results): artifacts · Stores predictions, tuning outputs, summaries, and ranking tables. · `data/results`
- [Model files](https://github.com/pauldasheinrich/idp/tree/master/models): artifacts · Persisted pickled estimators for real-only and augmented regimes. · `models`
- [Report](https://github.com/pauldasheinrich/idp/blob/master/reports/final_report.md): reporting · Markdown report and performance plots summarizing final results. · `reports/final_report.md`
- [Streamlit app](https://github.com/pauldasheinrich/idp/blob/master/frontend/streamlit_app.py): consumer UI · Reads pipeline outputs and defaults to present results without rerunning training. · `frontend/streamlit_app.py`

### Orchestration

Top-level runner and shared configuration boundaries.

- [Config](https://github.com/pauldasheinrich/idp/blob/master/config.json): settings · Central configuration for pipeline, features, targets, and model settings. · `config.json`
- [Pipeline runner](https://github.com/pauldasheinrich/idp/blob/master/run_pipeline.py): orchestrator · Top-level two-pass workflow that searches settings, finalizes augmentation, and trains final payloads. · `run_pipeline.py`

## Connections

- Raw logs → Preprocess: parse to clean · Raw device logs are converted into processed records.
- Preprocess → Wake state: derive state · Processed records feed wake/sleep state derivation.
- Wake state → Sleep labels: label nights · State information is used to build nightly sleep labels.
- Sleep labels → Evening windows: bound windows · Labels anchor the pre-bedtime evening window selection.
- Evening windows → Night features: engineer features · Window definitions constrain nightly feature extraction.
- Night features → Model data: real features · Engineered real-data features are assembled for modeling.
- Synthetic gen → Night bootstrap: mode · One branch generates synthetic nights via bootstrap.
- Synthetic gen → Participant synth: mode · One branch generates pseudo-participant synthetic data.
- Night bootstrap → Model data: augment train · Synthetic nights are merged into training-only model data.
- Participant synth → Model data: augment train · Synthetic participants are merged into training-only model data.
- Config → Pipeline runner: control · The runner reads the central configuration.
- Pipeline runner → Synthetic gen: finalize augmentation · The orchestration layer regenerates augmentation after selection.
- Pipeline runner → Model data: assemble sets · The runner materializes final training and evaluation tables.
- Model data → Train: train data · Model-ready tables feed estimator fitting.
- Model data → Evaluate: eval data · Fold-based splits feed evaluation and tuning.
- Train → GAM: fit · Training supports GAM models.
- Train → Random Forest: fit · Training supports random forest models.
- Train → XGBoost: fit · Training supports XGBoost models.
- Evaluate → Results: write metrics · Evaluation writes predictions, tuning tables, and summaries.
- Train → Model files: persist · Final estimators are serialized as pickled model artifacts.
- Results → Report: summarize · Reports and plots are generated from evaluation outputs.
- Results → Streamlit app: consume · The UI consumes saved results and defaults for presentation.
- Model files → Streamlit app: load models · The frontend can inspect persisted model artifacts if needed.
