How xu-ji/information-bottleneck works
This is research code for studying how information bottleneck relates to deep-learning generalization, with SWAG-based Bayesian model estimation. The main workflow is to generate or load datasets, train models while estimating or constraining information measures, save model and SWAG state, then compute compression and result metrics. The repository also contains SWAG training and evaluation experiments for ensembles, uncertainty, segmentation, curvature, and model width. Some experiment directories are represented by their paths because their individual files were not sampled; edges are limited to README statements and SOURCE INDEX references.
Main entry points: scripts Toy experiment runners · data.py Dataset loaders · swag.py SWAG posterior · run_swag.py SWAG training
Components 24
Information bottleneck
- Synthetic data
toy/util/data.py - Toy models
toy/util/model.py - Toy experiment runners
toy/scripts - IB metrics
toy/util/general.py - SWAG training helpers
toy/util/swag.py - CIFAR multi-seed
dnn/swag_repo/MI/main_multiseed.py - MI analysis
dnn/swag_repo/MI/util.py
SWAG framework
- Dataset loaders
dnn/swag_repo/swag/data.py - Network architectures
dnn/swag_repo/swag/models/preresnet.py - Model layers
dnn/swag_repo/swag/models/layers.py - Joint transforms
dnn/swag_repo/swag/models/joint_transforms.py - Training losses
dnn/swag_repo/swag/losses.py - SWAG posterior
dnn/swag_repo/swag/posteriors/swag.py
SWAG experiments
- SWAG training
dnn/swag_repo/experiments/train/run_swag.py - Ensembling
dnn/swag_repo/experiments/ensembling - Uncertainty
dnn/swag_repo/experiments/uncertainty - Segmentation
dnn/swag_repo/experiments/segmentation - Segmentation training utilities
dnn/swag_repo/experiments/segmentation/utils/training.py - ImageNet experiments
dnn/swag_repo/experiments/imagenet
Model analysis
- Compression metrics
toy/scripts - Gradient covariance
dnn/swag_repo/experiments/grad_cov/grad_cov_utils.py - Hessian eigenvalues
dnn/swag_repo/experiments/hessian_eigs - Model width
dnn/swag_repo/experiments/width
Other components
- Researcher
Connections 31
- Researcher → Toy experiment runners: runs experiments
- Researcher → CIFAR multi-seed: runs experiments
- Toy experiment runners → Synthetic data: loads data
toy/scripts/main_swag.py - Toy experiment runners → Toy models: uses models
toy/scripts/main_swag.py - Toy experiment runners → IB metrics: uses metrics
toy/scripts/main_swag.py - Toy experiment runners → SWAG training helpers: uses helpers
toy/scripts/main_swag.py - Toy experiment runners → MI analysis: uses MI analysis
toy/scripts/main_swag.py - Toy experiment runners → Compression metrics: computes results
- CIFAR multi-seed → MI analysis: uses analysis
dnn/swag_repo/MI/main_multiseed.py - CIFAR multi-seed → Dataset loaders: uses loaders
dnn/swag_repo/MI/main_multiseed.py - SWAG training → Dataset loaders: loads data
dnn/swag_repo/experiments/train/run_swag.py - SWAG training → Network architectures: selects model
dnn/swag_repo/experiments/train/run_swag.py - SWAG training → Training losses: selects loss
dnn/swag_repo/experiments/train/run_swag.py - SWAG training → SWAG posterior: uses posterior
dnn/swag_repo/experiments/train/run_swag.py - Network architectures → Model layers: uses layers
- Network architectures → Joint transforms: uses transforms
- Segmentation → Segmentation training utilities: uses utilities
dnn/swag_repo/experiments/segmentation/train.py - Segmentation training utilities → Dataset loaders: uses loaders
dnn/swag_repo/experiments/segmentation/train.py - Segmentation → Training losses: uses losses
dnn/swag_repo/experiments/segmentation/train.py - Segmentation → Network architectures: uses models
dnn/swag_repo/experiments/segmentation/train.py - Ensembling → Dataset loaders: uses loaders
dnn/swag_repo/experiments/ensembling/ensembling.py - Ensembling → Training losses: uses losses
dnn/swag_repo/experiments/ensembling/ensembling.py - Ensembling → SWAG posterior: uses posterior
dnn/swag_repo/experiments/ensembling/ensembling.py - Uncertainty → Dataset loaders: uses loaders
dnn/swag_repo/experiments/uncertainty/uncertainty.py - Uncertainty → Training losses: uses losses
dnn/swag_repo/experiments/uncertainty/uncertainty.py - Uncertainty → SWAG posterior: uses posterior
dnn/swag_repo/experiments/uncertainty/uncertainty.py - Gradient covariance → Dataset loaders: used by runner
- Hessian eigenvalues → SWAG posterior: may use posterior
- Model width → SWAG posterior: uses posterior
dnn/swag_repo/experiments/width/swag_2d_plane.py - Model width → Dataset loaders: loads data
dnn/swag_repo/experiments/width/swag_2d_plane.py - ImageNet experiments → SWAG posterior: uses posterior
dnn/swag_repo/experiments/imagenet/run_swag_imagenet.py