How ryanwangzf/pac-bayes-ib works
This repository is a research demonstration of PAC-Bayes Information Bottleneck (PAC-Bayes IB) methods for neural-network training and information-in-weights analysis. The principal workflow loads and preprocesses a dataset, initializes an MLP or VGG model and prior weights, trains while tracking information and accuracy, then evaluates or plots the results. Separate paths demonstrate IIW-regularized training and conventional VGG baselines. The README documents notebook experiments for phase transitions and batch-size/generalization analysis. Some notebook and implementation details were not sampled, so only documented or excerpt-supported connections are shown.
Main entry points: run_vgg_PIB.py PIB VGG experiment · utils.py Standard training · run_vgg_vanilla.py VGG baselines · dataset.py Dataset loaders
Components 20
Data preparation
- Dataset loaders
src/dataset.py - Preprocessing and metrics
src/utils.py
Neural models
- MLP model
src/models.py - VGG model
src/models.py - Information measure
src/models.py
Training and analysis
- Standard training
src/utils.py - Prior training
src/utils.py - Information tracking
src/utils.py - PAC-Bayes IB training
src/pib_utils.py - IIW regularization
src/pib_utils.py - Analysis plots
src/utils.py
Experiment entry points
- PIB VGG experiment
run_vgg_PIB.py - VGG baselines
run_vgg_vanilla.py - Dropout baseline
run_vgg_dropout.py - L2 baseline
run_vgg_l2.py - Notebook experiments
demo_track_information.ipynb - Batch-size analysis
demo_batch_size_info.ipynb - PAC-Bayes IB demo
demo_train_pac_bayes_ib.ipynb
Other components
- Researcher
- Datasets
Connections 29
- Researcher → PIB VGG experiment: runs
README.md - Researcher → Notebook experiments: explores
README.md - Researcher → Batch-size analysis: explores
README.md - Researcher → PAC-Bayes IB demo: explores
README.md - Datasets → Dataset loaders: provides data
src/dataset.py - Dataset loaders → Preprocessing and metrics: imports preprocessing
src/dataset.py - PIB VGG experiment → Dataset loaders: loads data
run_vgg_PIB.py - PIB VGG experiment → Preprocessing and metrics: preprocesses data
run_vgg_PIB.py - PIB VGG experiment → VGG model: initializes
run_vgg_PIB.py - PIB VGG experiment → Standard training: trains prior
run_vgg_PIB.py - PIB VGG experiment → PAC-Bayes IB training: starts PIB training
run_vgg_PIB.py - PIB VGG experiment → Preprocessing and metrics: evaluates results
run_vgg_PIB.py - PAC-Bayes IB training → Information measure: computes information
src/pib_utils.py - PAC-Bayes IB training → Preprocessing and metrics: evaluates accuracy
src/pib_utils.py - Standard training → Preprocessing and metrics: evaluates accuracy
src/utils.py - Prior training → Standard training: trains model
src/utils.py - Information tracking → Information measure: tracks information
README.md - Information tracking → Analysis plots: supplies metrics
README.md - PAC-Bayes IB demo → IIW regularization: demonstrates
README.md - Batch-size analysis → Information tracking: analyzes
README.md - Notebook experiments → Information tracking: demonstrates
README.md - VGG baselines → Dataset loaders: loads data
run_vgg_vanilla.py - VGG baselines → Preprocessing and metrics: preprocesses data
run_vgg_vanilla.py - VGG baselines → VGG model: initializes
run_vgg_vanilla.py - VGG baselines → Standard training: trains model
run_vgg_vanilla.py - Dropout baseline → VGG model: initializes dropout
run_vgg_dropout.py - Dropout baseline → Standard training: trains model
run_vgg_dropout.py - L2 baseline → VGG model: initializes
run_vgg_l2.py - L2 baseline → Standard training: trains with L2
run_vgg_l2.py