How pauldasheinrich/idp works
1. Overall shape
This repository is an end-to-end sleep modeling pipeline, not just a model package. It turns raw wearable/device logs into labeled nightly records, engineers predictors, generates synthetic augmentation data, trains/evaluates several regressors, and stores both final models and analysis outputs.
2. Raw data ingress
The starting point is data/raw/, which contains device log text files for a small set of participants. The src.data.raw_logs module is the first pipeline stage and is responsible for parsing those logs into a structured intermediate form.
3. Core preprocessing and labeling
The main real-data processing chain is:
raw_logs -> preprocess -> wake_state -> sleep_labels -> features
These stages live under src/data/ and progressively clean the logs, derive wake/sleep state, create sleep labels, and build nightly feature tables. The repository’s permanent real-data feature inputs are data/features/features_real.csv and data/windows/evening_windows.csv.
4. Feature boundary and leakage control
Feature engineering is deliberately constrained to pre-bedtime windows only, using bedtime_proxy - 4 hours to bedtime_proxy. Light is explicitly allowed only as a predictor, not as an input to wake/sleep label derivation. This is an important architectural boundary because the pipeline is designed to avoid label leakage.
5. Synthetic data augmentation subsystem
src/augmentation/generate.py produces synthetic training data under data/augmented/. There are two augmentation modes:
- real_plus_synthetic_nights: moving-block bootstrap over 10-minute evening windows
- real_plus_synthetic_participants: pseudo-participants with coherent synthetic nights, participant-level perturbations, temporal drift, and conservative SOL shifts
Synthetic artifacts are kept separate from real inputs, with outputs like synthetic_features.csv, synthetic_evening_windows.csv, and synthetic_participant_profiles.csv.
6. Model-ready dataset assembly
src.data.model_data combines the engineered real data and the chosen synthetic augmentation into training/evaluation tables. The repository shows both the final merged dataset in data/model/ and fold-specific splits in data/results/folds/. This is where the pipeline materializes training conditions such as real-only vs. real-plus-synthetic variants.
7. Evaluation design and split strategies
Evaluation is fold-based and supports participant and temporal holdouts. The data/results/folds/ directory contains fold-specific train/test CSVs for multiple scenarios, including:
- participant holdout
- temporal holdout
- per-participant folds
- global temporal holdout
For each fold, the pipeline creates separate training sets for real-only, real-plus-synthetic-nights, and real-plus-synthetic-participants conditions. Held-out test participants are never augmented.
8. Modeling subsystem
The modeling code lives in src/models/ and supports three model families:
- GAM
- Random Forest
- XGBoost
These are trained and evaluated under configurable feature sets, target transforms, and hyperparameter grids defined in config.json and consumed by src.models.config and related modules.
9. Experiment outputs and artifacts
Evaluation writes predictions, tuning results, summary tables, and rankings to data/results/. Important artifacts include:
- augmentation_model_results.csv
- augmentation_fold_predictions.csv
- augmentation_summary.csv
- tuning_trials.csv / tuning_best_params.csv
- augmentation_tuning_trials.csv / augmentation_tuning_best.csv
- model_rmse_ranking.csv
Model artifacts are persisted under models/ as pickled estimators, with separate files for real-only and augmented training regimes.
10. Pipeline orchestration
run_pipeline.py is the top-level orchestrator. It performs a two-pass workflow: first it searches augmentation/model settings and evaluates them, then it selects the best augmentation configuration, regenerates augmented data, reruns final evaluation, and trains the final model payloads. The --skip-finalize flag stops after the initial search/evaluation pass.
11. User-facing analysis and reporting
The repository includes a lightweight reporting layer in reports/ and plotting utilities such as plot_model_rmse.py. Final report content is stored as Markdown and figures summarize model performance metrics like RMSE, MAE, R², and error statistics.
12. Frontend boundary
frontend/streamlit_app.py is a separate Streamlit-based UI that reads pipeline outputs rather than recomputing the pipeline itself. It has its own requirements.txt and pipeline_defaults.json, so it functions as a consumer of the modeling outputs and default configuration, not as the core processing engine.
13. Development and provenance
The notebooks/ directory captures the same pipeline stages in exploratory form, mirroring the production modules. These notebooks document the analysis path but the production architecture is the Python module pipeline plus the orchestrating runner.
14. Key architectural constraint
The most important system boundary is between real data and synthetic data: synthetic rows are used only for training, never for held-out evaluation. Final evaluation is always on real participant data from data/model/model_table_real.csv, which keeps the performance estimates anchored to observed data.
Main entry points: model_data.py Model data · train.py Train · results Results · run_pipeline.py Pipeline runner
Components 21
Data prep
Raw logs, preprocessing, labels, and feature assembly for nightly sleep records.
- Raw logs · Parses wearable/device text logs into a structured intermediate form.
src/data/raw_logs.py - Preprocess · Cleans and standardizes the parsed logs into processed records.
src/data/preprocess.py - Wake state · Derives wake/sleep state used by downstream labeling.
src/data/wake_state.py - Sleep labels · Builds nightly sleep labels from processed state information.
src/data/sleep_labels.py - Night features · Creates pre-bedtime nightly feature tables with leakage-aware windows.
src/data/features.py - Evening windows · Defines pre-bedtime evening windows used for feature construction.
src/data/windows.py
Augmentation
Synthetic data generation and augmentation modes used for training only.
- Synthetic gen · Generates synthetic training data in nightly and participant-level modes.
src/augmentation/generate.py - Night bootstrap · Creates synthetic nights with moving-block bootstrap over evening windows.
src/augmentation/generate.py - Participant synth · Creates pseudo-participants with coherent synthetic nights and drift.
src/augmentation/generate.py
Modeling
Dataset assembly, training, tuning, and evaluation of regressors.
- Model data · Merges real features with selected augmentation into model-ready tables.
src/data/model_data.py - Train · Fits GAM, Random Forest, and XGBoost models under selected conditions.
src/models/train.py - Evaluate · Runs fold-based evaluation, tuning, and summary ranking.
src/models/evaluate.py - GAM · Generalized additive model implementation and training logic.
src/models/gam.py - Random Forest · Random forest regression implementation and training logic.
src/models/random_forest.py - XGBoost · XGBoost regression implementation and training logic.
src/models/xgboost.py
Outputs
Persisted results, models, reports, and the Streamlit consumer.
- Results · Stores predictions, tuning outputs, summaries, and ranking tables.
data/results - Model files · Persisted pickled estimators for real-only and augmented regimes.
models - Report · Markdown report and performance plots summarizing final results.
reports/final_report.md - Streamlit app · Reads pipeline outputs and defaults to present results without rerunning training.
frontend/streamlit_app.py
Orchestration
Top-level runner and shared configuration boundaries.
- Config · Central configuration for pipeline, features, targets, and model settings.
config.json - Pipeline runner · Top-level two-pass workflow that searches settings, finalizes augmentation, and trains final payloads.
run_pipeline.py
Connections 23
- Raw logs → Preprocess: parse to clean · Raw device logs are converted into processed records.
- Preprocess → Wake state: derive state · Processed records feed wake/sleep state derivation.
- Wake state → Sleep labels: label nights · State information is used to build nightly sleep labels.
- Sleep labels → Evening windows: bound windows · Labels anchor the pre-bedtime evening window selection.
- Evening windows → Night features: engineer features · Window definitions constrain nightly feature extraction.
- Night features → Model data: real features · Engineered real-data features are assembled for modeling.
- Synthetic gen → Night bootstrap: mode · One branch generates synthetic nights via bootstrap.
- Synthetic gen → Participant synth: mode · One branch generates pseudo-participant synthetic data.
- Night bootstrap → Model data: augment train · Synthetic nights are merged into training-only model data.
- Participant synth → Model data: augment train · Synthetic participants are merged into training-only model data.
- Config → Pipeline runner: control · The runner reads the central configuration.
- Pipeline runner → Synthetic gen: finalize augmentation · The orchestration layer regenerates augmentation after selection.
- Pipeline runner → Model data: assemble sets · The runner materializes final training and evaluation tables.
- Model data → Train: train data · Model-ready tables feed estimator fitting.
- Model data → Evaluate: eval data · Fold-based splits feed evaluation and tuning.
- Train → GAM: fit · Training supports GAM models.
- Train → Random Forest: fit · Training supports random forest models.
- Train → XGBoost: fit · Training supports XGBoost models.
- Evaluate → Results: write metrics · Evaluation writes predictions, tuning tables, and summaries.
- Train → Model files: persist · Final estimators are serialized as pickled model artifacts.
- Results → Report: summarize · Reports and plots are generated from evaluation outputs.
- Results → Streamlit app: consume · The UI consumes saved results and defaults for presentation.
- Model files → Streamlit app: load models · The frontend can inspect persisted model artifacts if needed.