How fchaubard/gradient_agreement_filtering works
Main Components
a) Core GAF Component:
• Two primary interfaces: step_GAF() and train_GAF()
• Gradient filtering mechanism using cosine distance
b) Integration Layer:
• How GAF interfaces with existing optimizers (SGD, Adam, AdamW, RMSProp)
• Demonstrate the connection to PyTorch's optimization framework
c) Data Flow Components:
• Input processing (model, optimizer, criterion, microbatches)
• Gradient computation and agreement filtering
• Output handling (filtered gradients, optimization steps)
Specific Design Elements
┌─────────────────────────────────────────────────────────┐
• External Interface │
• ┌─────────────┐ ┌──────────────┐ │
• step_GAF() │ │ train_GAF() │ │
• ┘ └──────────────┘ │
• ▼ ▼ │
• ┌─────────────────────────────────────────────────┐ │
• GAF Core Engine │ │
• ┌─────────────┐ ┌────────────────────┐ │ │
• Gradient │ │ Cosine Distance │ │ │
• Computing │◄─────► │ Calculator │ │ │
• ┘ └────────────────────┘ │ │
• ▼ ▼ │ │
• ┌─────────────────────────────────────────┐ │ │
• Agreement Filtering Logic │ │ │
• ┘ │ │
• ┘ │
• ▼ │
• ┌─────────────────────────────────────────────────┐ │
• Optimizer Interface │ │
• ┌────────┐ ┌─────┐ ┌───────┐ ┌────────┐ │ │
• SGD │ │Adam │ │AdamW │ │RMSProp │ │ │
• ┘ └─────┘ └───────┘ └────────┘ │ │
• ┘ │
• ▼ │
• ┌─────────────────────────────────────────────────┐ │
• Monitoring & Logging │ │
• ┌────────────────┐ ┌─────────────────┐ │ │
• Weights & Biases│ │ Verbose Logging │ │ │
• ┘ └─────────────────┘ │ │
• ┘ │
• ┘
Important Relationships
Input Data → Microbatches → Gradient Computation →
Agreement Filtering → Optimizer Update → Model Update
Notes for Implementation Details
• Annotations showing key parameters (cos_distance_thresh, num_batches_to_force_agreement)
• Version compatibility information
• Supported frameworks (PyTorch)
Remember to include a small textbox explaining that GAF is designed to bolt on top of existing optimization procedures, as this is a key feature of the system.