How meta-pytorch/opacus works
Project type and purpose
• Type: Open-source Python library (not a standalone service or web app).
• Purpose: Enable DP-SGD training in PyTorch via minimal code changes.
File structure
• Top-level “opacus” package: contains subpackages for core engine (privacy_engine.py), accountants (accountants/), gradient sampling (grad_sample/), DP layers (layers/), optimizers (optimizers/), schedulers (schedulers/), utility functions (utils/), and validators (validators/).
• Support files: Examples (examples/), Benchmarks (benchmarks/), Tutorials (tutorials/), Research experiments (research/), Documentation (docs/, website/).
• Configuration & CI: .github/workflows/, setup.py, requirements.txt, etc.
README for key insights
• Core concept: instantiate a PrivacyEngine, call make_private(model, optimizer, data_loader, …), then proceed with standard PyTorch training.
• Major abstractions:
• PrivacyEngine – the façade for making components “private.”
• Accountants – compute and track the privacy budget.
• Fast/ghost clipping – memory-efficient gradient clipping.
• Grad samplers – hooks into per-sample gradient computation.
Main components
a. API Layer
• User script / Jupyter notebook (examples/tutorials)
• Calls into PrivacyEngine
b. PrivacyEngine (core)
• Accepts: PyTorch Module, Optimizer, DataLoader
• Returns: wrapped versions with DP hooks
c. DP Hooks Pipeline
1. Gradient Sampling (grad_sample/): intercepts backward pass to compute per-sample gradients.
2. Clipping & Noise Scheduling (utils/fast_gradient_clipping_utils.py, schedulers/noise_scheduler.py): clip per-sample grads, add Gaussian noise.
3. Accountant (accountants/*): update privacy ledger (RDP, GDP, PRV analyses).
4. Optimizers (optimizers/*): wrappers around SGD/Adam that integrate clipped-noisy gradients.
d. DP Layers (layers/*): DP-aware versions of RNN, multihead attention, etc., used for built-in modules.
e. Validators (validators/*): pre-flight checks/fixes on user models to ensure compatibility with DP (e.g., no unsupported ops).
f. External Dependencies
• PyTorch core (autograd, optim, nn)
• NumPy (benchmarks, utils)
Relationships & interactions
• User script → PrivacyEngine
• PrivacyEngine → hooks into Model, Optimizer, DataLoader
• Model’s backward pass → GradientSampling → Clipping → NoiseScheduler → Accountant → Optimizer.update
• Accountant → exposes privacy budget metric back to user/UI
• That DP Layers and Validators plug into Model construction phase.
• Examples/tutorials/benchmarks drive end-to-end usage.
Architectural patterns & design principles
• Layered modularity: each concern (accounting, sampling, clipping, optimization) is in its own package.
• Plugin-style hooks: DP functionality is injected via wrappers rather than core PyTorch modifications.
• Extensibility: new accountants or clipping strategies can be added by dropping modules into the respective directories.