How andrewk1/pytorch-deep-bayesian-bandits works
Project Type and Purpose
• This is an open-source research tool—a Python/ PyTorch library and experimental framework—for “Deep Bayesian Contextual Bandits.”
• It integrates an unsupervised representation learner (a VAE) with contextual bandit algorithms (deep MLP + Thompson sampling) to run sequential decision-making experiments on datasets (MNIST, Mushroom, Yelp, etc.).
High-Level Architectural Overview
• The system is a monolithic ML pipeline composed of:
a. Data sources and dataset management
b. Data loading & preprocessing
c. Unsupervised representation learning (VAE)
d. Contextual bandit engine (MLP + sampling algorithms)
e. Experiment orchestration & result storage
f. Interactive analysis via Jupyter notebooks
File-Structure Analysis
• Top-level scripts:
• run_experiment*.py – command-line entry points (single-threaded, multithreaded, dataset-specific)
• bandits/ package:
• core/ – definitions of ContextualBandit, ContextualDataset, Hyperparameter containers
• data/ – data extraction, sampling, caching VAE features, dataset readers (MNIST, Mushroom, raw stock contexts)
• algorithms/ – neural_bandit_model (MLP encoder), neural_linear_sampling (Thompson sampling logic)
• vae/ package:
• model/ – VAE architecture code
• results/ – serialized VAE outputs and embeddings (vae_features.npy)
• vae.ipynb – interactive notebook to train/explore the VAE
• Notebooks at root: prototyping data loading, baseline models, paragraph encodings (BERT).
Key Architectural Patterns & Principles
• Modular separation by concern:
• Data ingestion & sampling
• Unsupervised feature learning
• Contextual bandit decision logic
• Experiment control & hyperparameter management
• Clear layer of abstraction: VAE embeddings → Bandit MLP input
• Configurability via hyperparams objects and command-line scripts
• Optional multithreading for parallel experiment runs
Core Components
A. Data Layer
• Raw datasets (bandits/data/selected_datasets, raw mushroom, MNIST)
• Data extractor & sampler modules
B. Preprocessing & Feature Cache
• cache_vae_features.py → generates/loads vae_features.npy
C. Unsupervised Learning (VAE)
• vae.model → Encoder/Decoder networks
• vae training pipeline (vae.ipynb & scripts)
D. Contextual Bandit Engine
• MLP contextual encoder (neural_bandit_model.py)
• Thompson sampling logic (neural_linear_sampling.py)
• Core abstractions (ContextualBandit, ContextualDataset)
E. Experiment Orchestrator
• run_experiment*.py – wiring data, VAE embeddings, bandit engine, logging results
• Hyperparameter loader (hyperparams.py)
F. Output & Analysis
• Results directory (vae/results, experiment logs)
• Jupyter notebooks for post-hoc analysis
Relationships & Data Flow
• Embeddings feed into ContextualDataset → MLP Bandit Engine
• Bandit Engine outputs decisions & rewards → logged by Experiment Orchestrator
• Results archived → notebooks for visualization
Technologies & Frameworks
• Python 3.x, PyTorch for all neural modules
• NumPy for data handling
• Jupyter Notebooks for prototyping/analysis
• Optional Python multiprocessing for multithreaded runs