How mscherbela/kfac-jax works
Project Type and Purpose
• KFAC-JAX is an open-source Python library (not a standalone application) providing second-order optimization (K-FAC) and scalable curvature estimation for neural networks built on top of JAX.
• Its primary consumers are researchers and engineers who write model code in JAX (often with frameworks like Haiku) and want to plug in a drop-in optimizer.
File-Structure Analysis
Top-level directories and files:
• .github/, CI workflows: automation for testing and publishing.
• README.md, LICENSE, CONTRIBUTING.md, setup.py, requirements*.txt: packaging, installation, contribution guidelines.
• docs/: Sphinx source for user/documentation site.
• examples/: concrete end-to-end scripts illustrating how to use the library.
• tests/: unit tests for each core component.
• kfac_jax/_src/: library source code, broken into modules:
– optimizer.py – public Optimizer class and step logic
– curvature_estimator.py – driver for curvature approximation
– curvature_blocks.py – per-layer block implementations (dense, conv, etc.)
– layers_and_loss_tags.py, loss_functions.py – loss registration and tagging
– tracer.py, tag_graph_matcher.py – JAX graph introspection to match blocks to computation
– patches_second_moment.py – low-level moment computations
– utils/ – submodules for math, accumulators, staging, parallel utilities, types
1. User Model & Training Script (examples or user code)
2. Public API Layer (kfac_jax/__init__.py)
3. Optimizer Module (optimizer.py)
4. Curvature Estimator (curvature_estimator.py)
5. Curvature Blocks (curvature_blocks.py)
6. Loss & Layer Registration (layers_and_loss_tags.py, loss_functions.py)
7. Tracer & Graph Matcher (tracer.py, tag_graph_matcher.py)
8. Patches & Low-Level Ops (patches_second_moment.py)
9. Utilities (utils/…)
10. External Dependencies:
– JAX (XLA backend, JIT, pmap)
– Haiku (or other NN frameworks)
– Python environment
1 → 2: User code imports kfac_jax API.
2 → 3: Optimizer is instantiated with user’s loss and gradient function.
3 ↔ 4: Optimizer calls CurvatureEstimator inside each step.
4 → 5: Estimator delegates per-layer curvature computation to the appropriate block.
4 ← 6: Before stepping, the registration subsystem has tagged parameters/losses so estimator knows what to do.
5 ← 7: Blocks are selected by matching the JAX computation graph via tracer + graph matcher.
5 ↔ 8: Blocks use low-level “second moment” patches for efficient moment updates.
3,4,5 → 9: All modules use common math, accumulators, parallel, staging utilities.
Everything relies on JAX primitives (jit, vmap, pmap, XLA) under the hood.
Architectural Patterns & Design Principles
• Modular layering: clear separation of public API, orchestration (optimizer), estimation, block implementations, and utilities.
• Strategy/factory pattern: different curvature blocks selected at runtime based on tags.
• Observer/tracer: introspects JAX computation graph to register and match layers.
• Registration system: declarative tagging of losses and parameters.
• Single responsibility for each module: easier to test (see tests/) and extend with new blocks or loss functions.