How ubc-systopia/dp-adambc works
Project type and purpose
• This repository implements Differentially Private optimizers (DP-AdamBC, DP-Adam, DP-SGD) and compares them on vision (CIFAR10) and NLP (GLUE) tasks.
• It consists of a core Python pipeline (in the “GNN” directory) plus two parallel reference implementations: one in JAX and one in PyTorch.
File structure
Top-level modules:
– GNN/ ← core data/model/training framework, privacy accountants & hyperparameter tuning
– jax_implementation/ ← JAX-based pipeline: data_utils, models, trainers
– pytorch_implementation/ ← PyTorch-based pipeline: configlib, Opacus integration for text classification
– README.md ← usage examples, DH-AdamBC paper reference
Inside GNN/:
– main.py, train.py ← orchestration of training runs
– dataset_readers.py + download_datasets.py + input_pipeline.py ← data ingestion & batching
– models.py + normalizations.py ← model definitions & preprocessing
– optim/, optimizers.py + adam_optimizers.py ← DP optimizers, optimizer interface
– privacy_accountants.py ← ε accounting
– sampler.py ← sampling logic for privacy
– autotune/
• best_params/ + SQL scripts + tune_adambc_example.json ← hyperparameter-tuning subsystem
– configs/ ← named experiment configurations
– *_test.py ← unit tests
– run_tuning.sh, requirements.txt ← tooling & dependencies
README for additional insights
• Shows how to invoke:
– Core (GNN) or Top-level main.py for vision tasks (CNN5 on CIFAR10)
– jax_implementation/main.py for DP-AdamBC via JAX
– pytorch_implementation/text_classification_glue.py or opacus_text_classification.py for NLP
a. Main components
• Data Layer
– Raw dataset download & storage
– Dataset readers & input pipeline (batching, augmentation)
• Core DP Framework (“GNN”)
– Models & normalizations
– Sampler (per-step sampling for DP)
– Optimizer interface & DP algorithms (DPAdam, DPIterative/DP-SGD)
– Privacy accountant (tracks cumulative ε)
– Hyperparameter tuner (Autotune subsystem)
– Experiment configs
– Orchestrator (main.py / train.py)
• Implementation Tracks
– JAX pipeline: data_utils → model modules → trainer modules → utils
– PyTorch pipeline: configlib → text classifiers → Opacus wrappers
• External Services
– SQL (for storing/retrieving best hyperparams)
– JSON config files
b. Relationships & interactions
– Data Layer feeds into both Core framework and two implementation tracks
– Orchestrator reads a config → instantiates dataset pipeline & model → selects trainer & optimizer → runs training loop
– During training, optimizer interacts with sampler and privacy accountant on each step
– Hyperparameter tuner invokes multiple orchestrator runs, logs results to SQL/JSON
– Implementation tracks mirror this flow but use framework-specific primitives
c. Architectural patterns & principles
– Modular “plugin” architecture: optimizers, trainers, models each follow a common interface
– Separation of concerns between data ingestion, model definition, optimization, and privacy accounting
– Extensible tuning subsystem decoupled from training logic via scripts and JSON specs
d. Key technologies & libraries
– Python (core)
– JAX & Flax (jax_implementation)
– PyTorch & Opacus (pytorch_implementation)
– SQL database or SQLite (autotune)
– JSON for hyperparameter specs
– Command-line args (via argparse / configlib)