How pytorch/captum works
Project Type and Purpose
• Captum is an open-source Python library for model interpretability on top of PyTorch.
• It provides a modular collection of attribution methods (IntegratedGradients, Saliency, etc.), influence-based methods (TracIn), concept analysis (TCAV), adversarial robustness tools, and a web/UI visualization layer called Captum Insights.
High-Level Components
A. Core Python Package (captum/)
• attr – all attribution algorithms; subfolders for _core implementations (including saliency), _models, and _utils.
• concept – concept activation vector algorithms (TCAV).
• influence – influence functions and related utilities.
• metrics – infidelity and sensitivity metrics.
• module – stochastic gate modules.
• robust – adversarial attack and perturbation algorithms.
• insights – server and Jupyter widget; includes a Flask/FastAPI-style Python server and React frontend under attr_vis/frontend.
B. Tutorials & Examples (tutorials/) – Jupyter notebooks and data for hands-on demos.
C. Documentation (docs/, sphinx/) – static site generation source.
D. Website (website/) – Docusaurus/React site for marketing and docs hosting.
E. CI/CD & Build Scripts (scripts/, .github/) – test pipelines, linting, packaging, documentation deployment.
F. External Dependencies – PyTorch core, TorchVision/TorchText adapters, Node.js/Yarn for building Insights frontend.
Saliency Focus
• The Saliency method lives in captum/attr/_core/saliency.py and uses gradient utilities in captum/_utils/gradient.py and sample_gradient.py.
• It depends on the common framework in captum/attr/_core/common.py to handle inputs, baselines, target selection, and convergence deltas.
Relationships & Data Flow
1. User model (PyTorch nn.Module) + input tensor
↓
2. Captum Saliency component (captum.attr._core.saliency)
• Queries gradient utilities → computes gradient of output w.r.t. inputs
• Returns attribution map & optional convergence metrics
↓
3. Higher-level utilities (e.g., NoiseTunnel, LayerSaliency) can wrap Saliency for smoothing or internal layer attribution
↓
4. Optional visualization:
• Captum Insights server → REST or WebSocket API → React frontend widget
• Jupyter widget embedding via captum.insights.attr_vis.widget
Architectural Patterns & Design Principles
• Modular plugin-style: Each algorithm in its own module under attr/_core.
• Separation of concerns: Core compute layer (algorithms) decoupled from visualization/UI layer.
• Reusability: Common utilities for batching, gradient sampling, typing.
• Extensibility: Adding a new algorithm involves creating a new _core/<algo>.py and registering it with the top-level captum.attr namespace.