How google-deepmind/dm-haiku works
Project type and purpose
• This is a Python neural-network library for JAX, akin to “Sonnet for JAX.”
• It provides an object-oriented module abstraction (hk.Module), parameter/state management, RNG sequencing, and functional transforms (hk.transform, hk.transform_with_state) that integrate seamlessly with JAX’s jit, grad, pmap, etc.
• The repo is structured to support the core library (haiku/_src), examples, documentation, benchmarks, tests, CI/CD, and packaging.
File structure
• Top-level: configuration files (.pylintrc, .style.yapf), build scripts (BUILD, Bazel files), CI workflows, setup.py, requirements*.txt.
• haiku/ package:
• _src/: the core implementation, divided into modules for layers, transforms, state/RNG management, integration tests, nets, utils.
• benchmarks/, experimental/, top-level convenience modules (haiku/module.py, haiku/testing.py, etc.)
• examples/: runnable demos (MNIST, ResNet, IMPALA RL), plus BUILD files for Bazel.
• docs/: Sphinx-based docs, notebooks, custom extensions.
• Tests sprinkled alongside source (*_test.py) and integration checkpoints (.json fixtures).
README for additional insights
• Core abstractions:
a. hk.Module for OO layering and parameter/state references
b. hk.transform / hk.without_apply_rng for converting impure OO functions into pure (init, apply) pairs
c. hk.transform_with_state for tracking mutable state (e.g. batch-norm moving averages)
• Emphasis on lifting a user-written function into pure init/apply, then lowering parameters and rng back into the forward pass.
• Examples of distributed training via jax.pmap.
Detailing the lifting/lowering between functional and OO
• Lifting (OO → Functional): annotate how hk.transform traverses the Python AST (or via decorators) to intercept calls to hk.get_parameter, hk.get_state, hk.next_rng_key. It builds up a graph of requested parameters and state, and returns two side-effect-free functions:
• apply: replay with the collected params & state, feeding RNG from a seeded sequence
• Lowering (Functional → OO execution): show how the apply function dispatches calls back into the module’s __call__ methods with the parameter values injected. The RNG manager supplies fresh keys per module invocation, preserving reproducibility.