# xu-ji/information-bottleneck architecture

> Architecture diagram of the GitHub repository xu-ji/information-bottleneck, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xu-ji/information-bottleneck
- Repository: https://github.com/xu-ji/information-bottleneck

## Overview

This is research code for studying how information bottleneck relates to deep-learning generalization, with SWAG-based Bayesian model estimation. The main workflow is to generate or load datasets, train models while estimating or constraining information measures, save model and SWAG state, then compute compression and result metrics. The repository also contains SWAG training and evaluation experiments for ensembles, uncertainty, segmentation, curvature, and model width. Some experiment directories are represented by their paths because their individual files were not sampled; edges are limited to README statements and SOURCE INDEX references.

Main entry points: [`toy/scripts`](https://github.com/xu-ji/information-bottleneck/tree/main/toy/scripts) (Toy experiment runners), [`dnn/swag_repo/swag/data.py`](https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/swag/data.py) (Dataset loaders), [`dnn/swag_repo/swag/posteriors/swag.py`](https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/swag/posteriors/swag.py) (SWAG posterior), [`dnn/swag_repo/experiments/train/run_swag.py`](https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/experiments/train/run_swag.py) (SWAG training)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_toy["Information bottleneck"]
  node_toy_data["Synthetic data<br/>[data.py]"]
  node_toy_models["Toy models<br/>[model.py]"]
  node_toy_training["Toy experiment runners"]
  node_ib_metrics["IB metrics<br/>[general.py]"]
  node_toy_swag["SWAG training helpers<br/>[swag.py]"]
  node_mi_runner["CIFAR multi-seed<br/>[main_multiseed.py]"]
  node_mi_utils["MI analysis<br/>[util.py]"]
end

subgraph group_swag["SWAG framework"]
  node_dnn_data["Dataset loaders<br/>[data.py]"]
  node_dnn_models["Network architectures<br/>[preresnet.py]"]
  node_model_layers["Model layers<br/>[layers.py]"]
  node_joint_transforms["Joint transforms"]
  node_losses["Training losses<br/>[losses.py]"]
  node_swag_posterior["SWAG posterior<br/>[swag.py]"]
end

subgraph group_experiments["SWAG experiments"]
  node_train_runner["SWAG training<br/>[run_swag.py]"]
  node_ensemble["Ensembling"]
  node_uncertainty["Uncertainty"]
  node_segmentation["Segmentation"]
  node_seg_training["Segmentation training utilities<br/>[training.py]"]
  node_imagenet["ImageNet experiments"]
end

subgraph group_analysis["Model analysis"]
  node_toy_results["Compression metrics"]
  node_grad_cov["Gradient covariance<br/>[grad_cov_utils.py]"]
  node_hessian["Hessian eigenvalues"]
  node_width["Model width"]
end

node_researcher(("Researcher"))

node_researcher -->|"runs experiments"| node_toy_training
node_researcher -->|"runs experiments"| node_mi_runner
node_toy_training -->|"loads data"| node_toy_data
node_toy_training -->|"uses models"| node_toy_models
node_toy_training -->|"uses metrics"| node_ib_metrics
node_toy_training -->|"uses helpers"| node_toy_swag
node_toy_training -->|"uses MI analysis"| node_mi_utils
node_toy_training -->|"computes results"| node_toy_results
node_mi_runner -->|"uses analysis"| node_mi_utils
node_mi_runner -->|"uses loaders"| node_dnn_data
node_train_runner -->|"loads data"| node_dnn_data
node_train_runner -->|"selects model"| node_dnn_models
node_train_runner -->|"selects loss"| node_losses
node_train_runner -->|"uses posterior"| node_swag_posterior
node_dnn_models -->|"uses layers"| node_model_layers
node_dnn_models -->|"uses transforms"| node_joint_transforms
node_segmentation -->|"uses utilities"| node_seg_training
node_seg_training -->|"uses loaders"| node_dnn_data
node_segmentation -->|"uses losses"| node_losses
node_segmentation -->|"uses models"| node_dnn_models
node_ensemble -->|"uses loaders"| node_dnn_data
node_ensemble -->|"uses losses"| node_losses
node_ensemble -->|"uses posterior"| node_swag_posterior
node_uncertainty -->|"uses loaders"| node_dnn_data
node_uncertainty -->|"uses losses"| node_losses
node_uncertainty -->|"uses posterior"| node_swag_posterior
node_grad_cov -.->|"used by runner"| node_dnn_data
node_hessian -.->|"may use posterior"| node_swag_posterior
node_width -->|"uses posterior"| node_swag_posterior
node_width -->|"loads data"| node_dnn_data
node_imagenet -->|"uses posterior"| node_swag_posterior

click node_toy_data "https://github.com/xu-ji/information-bottleneck/blob/main/toy/util/data.py"
click node_toy_models "https://github.com/xu-ji/information-bottleneck/blob/main/toy/util/model.py"
click node_toy_training "https://github.com/xu-ji/information-bottleneck/tree/main/toy/scripts"
click node_ib_metrics "https://github.com/xu-ji/information-bottleneck/blob/main/toy/util/general.py"
click node_toy_swag "https://github.com/xu-ji/information-bottleneck/blob/main/toy/util/swag.py"
click node_toy_results "https://github.com/xu-ji/information-bottleneck/tree/main/toy/scripts"
click node_mi_runner "https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/MI/main_multiseed.py"
click node_mi_utils "https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/MI/util.py"
click node_dnn_data "https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/swag/data.py"
click node_dnn_models "https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/swag/models/preresnet.py"
click node_model_layers "https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/swag/models/layers.py"
click node_joint_transforms "https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/swag/models/joint_transforms.py"
click node_losses "https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/swag/losses.py"
click node_swag_posterior "https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/swag/posteriors/swag.py"
click node_train_runner "https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/experiments/train/run_swag.py"
click node_ensemble "https://github.com/xu-ji/information-bottleneck/tree/main/dnn/swag_repo/experiments/ensembling"
click node_uncertainty "https://github.com/xu-ji/information-bottleneck/tree/main/dnn/swag_repo/experiments/uncertainty"
click node_segmentation "https://github.com/xu-ji/information-bottleneck/tree/main/dnn/swag_repo/experiments/segmentation"
click node_seg_training "https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/experiments/segmentation/utils/training.py"
click node_grad_cov "https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/experiments/grad_cov/grad_cov_utils.py"
click node_hessian "https://github.com/xu-ji/information-bottleneck/tree/main/dnn/swag_repo/experiments/hessian_eigs"
click node_width "https://github.com/xu-ji/information-bottleneck/tree/main/dnn/swag_repo/experiments/width"
click node_imagenet "https://github.com/xu-ji/information-bottleneck/tree/main/dnn/swag_repo/experiments/imagenet"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_toy_data,node_toy_models,node_toy_training,node_ib_metrics,node_toy_swag,node_mi_runner,node_mi_utils toneBlue
class node_dnn_data,node_dnn_models,node_model_layers,node_joint_transforms,node_losses,node_swag_posterior toneAmber
class node_train_runner,node_ensemble,node_uncertainty,node_segmentation,node_seg_training,node_imagenet toneMint
class node_toy_results,node_grad_cov,node_hessian,node_width toneRose
class node_researcher toneIndigo
```

## Components

### Information bottleneck

- [Synthetic data](https://github.com/xu-ji/information-bottleneck/blob/main/toy/util/data.py): `toy/util/data.py`
- [Toy models](https://github.com/xu-ji/information-bottleneck/blob/main/toy/util/model.py): `toy/util/model.py`
- [Toy experiment runners](https://github.com/xu-ji/information-bottleneck/tree/main/toy/scripts): `toy/scripts`
- [IB metrics](https://github.com/xu-ji/information-bottleneck/blob/main/toy/util/general.py): `toy/util/general.py`
- [SWAG training helpers](https://github.com/xu-ji/information-bottleneck/blob/main/toy/util/swag.py): `toy/util/swag.py`
- [CIFAR multi-seed](https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/MI/main_multiseed.py): `dnn/swag_repo/MI/main_multiseed.py`
- [MI analysis](https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/MI/util.py): `dnn/swag_repo/MI/util.py`

### SWAG framework

- [Dataset loaders](https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/swag/data.py): `dnn/swag_repo/swag/data.py`
- [Network architectures](https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/swag/models/preresnet.py): `dnn/swag_repo/swag/models/preresnet.py`
- [Model layers](https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/swag/models/layers.py): `dnn/swag_repo/swag/models/layers.py`
- [Joint transforms](https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/swag/models/joint_transforms.py): `dnn/swag_repo/swag/models/joint_transforms.py`
- [Training losses](https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/swag/losses.py): `dnn/swag_repo/swag/losses.py`
- [SWAG posterior](https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/swag/posteriors/swag.py): `dnn/swag_repo/swag/posteriors/swag.py`

### SWAG experiments

- [SWAG training](https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/experiments/train/run_swag.py): `dnn/swag_repo/experiments/train/run_swag.py`
- [Ensembling](https://github.com/xu-ji/information-bottleneck/tree/main/dnn/swag_repo/experiments/ensembling): `dnn/swag_repo/experiments/ensembling`
- [Uncertainty](https://github.com/xu-ji/information-bottleneck/tree/main/dnn/swag_repo/experiments/uncertainty): `dnn/swag_repo/experiments/uncertainty`
- [Segmentation](https://github.com/xu-ji/information-bottleneck/tree/main/dnn/swag_repo/experiments/segmentation): `dnn/swag_repo/experiments/segmentation`
- [Segmentation training utilities](https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/experiments/segmentation/utils/training.py): `dnn/swag_repo/experiments/segmentation/utils/training.py`
- [ImageNet experiments](https://github.com/xu-ji/information-bottleneck/tree/main/dnn/swag_repo/experiments/imagenet): `dnn/swag_repo/experiments/imagenet`

### Model analysis

- [Compression metrics](https://github.com/xu-ji/information-bottleneck/tree/main/toy/scripts): `toy/scripts`
- [Gradient covariance](https://github.com/xu-ji/information-bottleneck/blob/main/dnn/swag_repo/experiments/grad_cov/grad_cov_utils.py): `dnn/swag_repo/experiments/grad_cov/grad_cov_utils.py`
- [Hessian eigenvalues](https://github.com/xu-ji/information-bottleneck/tree/main/dnn/swag_repo/experiments/hessian_eigs): `dnn/swag_repo/experiments/hessian_eigs`
- [Model width](https://github.com/xu-ji/information-bottleneck/tree/main/dnn/swag_repo/experiments/width): `dnn/swag_repo/experiments/width`

### Other components

- **Researcher**

## Connections

- Researcher → Toy experiment runners: runs experiments
- Researcher → CIFAR multi-seed: runs experiments
- Toy experiment runners → Synthetic data: loads data (evidence: `toy/scripts/main_swag.py`)
- Toy experiment runners → Toy models: uses models (evidence: `toy/scripts/main_swag.py`)
- Toy experiment runners → IB metrics: uses metrics (evidence: `toy/scripts/main_swag.py`)
- Toy experiment runners → SWAG training helpers: uses helpers (evidence: `toy/scripts/main_swag.py`)
- Toy experiment runners → MI analysis: uses MI analysis (evidence: `toy/scripts/main_swag.py`)
- Toy experiment runners → Compression metrics: computes results
- CIFAR multi-seed → MI analysis: uses analysis (evidence: `dnn/swag_repo/MI/main_multiseed.py`)
- CIFAR multi-seed → Dataset loaders: uses loaders (evidence: `dnn/swag_repo/MI/main_multiseed.py`)
- SWAG training → Dataset loaders: loads data (evidence: `dnn/swag_repo/experiments/train/run_swag.py`)
- SWAG training → Network architectures: selects model (evidence: `dnn/swag_repo/experiments/train/run_swag.py`)
- SWAG training → Training losses: selects loss (evidence: `dnn/swag_repo/experiments/train/run_swag.py`)
- SWAG training → SWAG posterior: uses posterior (evidence: `dnn/swag_repo/experiments/train/run_swag.py`)
- Network architectures → Model layers: uses layers
- Network architectures → Joint transforms: uses transforms
- Segmentation → Segmentation training utilities: uses utilities (evidence: `dnn/swag_repo/experiments/segmentation/train.py`)
- Segmentation training utilities → Dataset loaders: uses loaders (evidence: `dnn/swag_repo/experiments/segmentation/train.py`)
- Segmentation → Training losses: uses losses (evidence: `dnn/swag_repo/experiments/segmentation/train.py`)
- Segmentation → Network architectures: uses models (evidence: `dnn/swag_repo/experiments/segmentation/train.py`)
- Ensembling → Dataset loaders: uses loaders (evidence: `dnn/swag_repo/experiments/ensembling/ensembling.py`)
- Ensembling → Training losses: uses losses (evidence: `dnn/swag_repo/experiments/ensembling/ensembling.py`)
- Ensembling → SWAG posterior: uses posterior (evidence: `dnn/swag_repo/experiments/ensembling/ensembling.py`)
- Uncertainty → Dataset loaders: uses loaders (evidence: `dnn/swag_repo/experiments/uncertainty/uncertainty.py`)
- Uncertainty → Training losses: uses losses (evidence: `dnn/swag_repo/experiments/uncertainty/uncertainty.py`)
- Uncertainty → SWAG posterior: uses posterior (evidence: `dnn/swag_repo/experiments/uncertainty/uncertainty.py`)
- Gradient covariance → Dataset loaders: used by runner
- Hessian eigenvalues → SWAG posterior: may use posterior
- Model width → SWAG posterior: uses posterior (evidence: `dnn/swag_repo/experiments/width/swag_2d_plane.py`)
- Model width → Dataset loaders: loads data (evidence: `dnn/swag_repo/experiments/width/swag_2d_plane.py`)
- ImageNet experiments → SWAG posterior: uses posterior (evidence: `dnn/swag_repo/experiments/imagenet/run_swag_imagenet.py`)
