How neuromorphiccomputing/softhebb works
Project Type and Purpose
• This is an open-source Python codebase for reproducing and extending the SoftHebb unsupervised Hebbian deep-learning algorithm (ICLR 2023 / Neuromorphic Comput. Eng. 2022).
Main Components
A. Entry-point Scripts
• demo.py: quick start for training and evaluation
• train.py / multi_layer.py: orchestrate training of one model across epochs
• ray_search.py: hyperparameter optimization harness using Ray Tune
• post_hoc_loss.py: compute post-hoc loss analyses
• graphs.ipynb: generate plots and figures
B. Configuration
• presets.json (or preset.json): defines network architectures, datasets, layer hyperparameters
• environment_pytorch==1.7.1.yml: conda env specification
C. Data and Dataset Management
• dataset.py: interfaces to download / load MNIST, CIFAR, ImageNette, ImageNet
• utils.py: global constants (DATA path), shared helper functions
D. Model Definition
• layer.py, hebblinear.py, hebbconv.py, multi_layer.py (model wrapper)
• activation.py: custom activation layers (soft winner-take-all, RePU, etc.)
• model.py: load/save checkpoint and define full multilayer models
E. Training Engine
• engine.py: one-epoch train/eval logic for a single model
• train.py: multi-epoch training loops for different modes (unsup, hybrid, supervised)
• log.py: logging facility for metrics and convergence stats
F. Hyperparameter Search
• ray_search.py: defines search space, runner logic, TensorBoard output
G. Results & Outputs
• DATA/hebb/results: models, per-layer checkpoints, network checkpoints
• DATA/hebb/search: Ray Tune logs for visualization
Data Flow & Interactions
1. User invokes a script (demo.py, multi_layer.py or ray_search.py) from CLI
2. Script reads preset.json to build a sequence of blocks:
• Blocks = {Flatten, BatchNorm, Convolutional or Fully-connected + Activation + Pooling}
3. dataset.py + utils.py load and preprocess data from DATA/dataset/…
4. model.py / multi_layer.py instantiate the PyTorch model according to presets
5. engine.py applies per-epoch training:
• For Hebbian layers: local weight updates (hebblinear/hebbconv rules)
• Optionally supervised BP on classifier block
6. log.py captures and persists metrics and layer statistics to disk
7. ray_search.py spins up multiple trials in parallel via Ray, writes TensorBoard logs under DATA/hebb/search
8. After training, graphs.ipynb / nb_utils.py produce figures for ICLR / paper
Architectural Patterns & Design Principles
• Modular “block” pattern: each network is a pipeline of interchangeable blocks (MLP or CNN blocks) configured purely via JSON presets.
• Separation of concerns:
• Data loading (dataset.py)
• Model definition (layer.py, model.py)
• Training logic (engine.py, train.py)
• Orchestration / CLI (multi_layer.py, ray_search.py)
• Visualization (graphs.ipynb)
Final Recommendations
• Every major directory/file grouping as a logical subsystem.
• Pipeline nature: presets.json drives everything.
• Verify that connections represent the actual Python imports and data dependencies.
• Keep it detailed but uncluttered: group low-level functions (utils, activations) under “Utilities” if needed.