How google-deepmind/penzai works
Project type and purpose
• Type: Python library (open-source toolkit) built on JAX.
• Purpose:
– Represent neural networks as functional pytree data structures.
– Expose a declarative combinator-based NN API, a pretty-printer (Treescope), and model-surgery “toolshed” scripts.
File structure
• Top-level directories:
– penzai/ ← the main source package
– docs/ ← Sphinx documentation sources
– notebooks/ ← tutorial and demo notebooks
– tests/ ← unit and integration tests
– .github/, CI files ← CI pipelines (unittests.yml)
– pyproject.toml ← packaging/build config
• Inside penzai/: logical subpackages:
– core/ ← tree and array utilities (selectors, named_axes, partitioning, random_stream, struct, shapecheck)
– nn/ ← declarative neural-network library (layers, combinators, parameters, layer_stack)
– models/ ← reference model definitions (simple_mlp; transformer variants: Gemma, GPT-NeoX, Llama, Mistral)
– pz/ ← alias namespace (pz.nn, pz.ts)
– toolshed/ ← utility scripts for training, checkpointing, visualization, model re-wiring
– treescope/ ← interactive pretty-printing (plugin for IPython/Colab)
– deprecated/v1/ ← legacy API for backwards compatibility
– experimental/v2/ ← early-stage API prototypes
README for additional insights
• Four “modules” highlighted:
1. Treescope (penzai.treescope, pz.ts)
2. Core utilities (penzai.core, pz.select, pz.nx)
3. Declarative NN (penzai.nn, pz.nn)
4. Transformer models (penzai.models.transformer)
• Integration points:
– User code (Jupyter/IPython) ↔ Treescope for visualization
– pz.select and pz.nx ↔ tree-traversal and named-axis APIs
– pz.nn layers ↔ core utilities for parameter management and stateful leaves
– Models built on top of pz.nn + pz.nx + pz.select
Main components & their relationships
A. External Dependencies (bottom layer)
– JAX (jax.numpy, jax.random, jax.tree_util)
– IPython/Colab display system
B. Core Package (penzai.core)
– Selectors module (pz.select)
– Named-axes module (pz.nx)
– Tree utilities, partitioning, random_stream, struct, shapecheck
C. Declarative Neural Net Library (penzai.nn, alias pz.nn)
– Layer base classes, Elementwise, combinators, linear/affine, dropout, embeddings, grouping, layer_stack
– Uses core utilities for mutable state, parameter sharing
D. Transformer & Model Package (penzai.models.transformer, penzai.models.simple_mlp)
– Model parts, sampling loops, pre-trained weights loading
– Uses pz.nn + pz.nx + pz.select
E. Treescope Visualization (penzai.treescope, alias pz.ts)
– Hooks into IPython/Colab display
– Renders arbitrary JAX pytrees and NDArrays
F. Toolshed Scripts (penzai.toolshed)
– Training helpers, gradient checkpointing, model rewiring, unflaxify, etc.
– Operate as standalone scripts or importable utilities
G. Deprecated & Experimental APIs
H. Documentation & Notebooks (side panel)
– docs/ ← Sphinx
– notebooks/ ← interactive demos
– Core ← calls/invokes ← External JAX APIs
– pz.nn ← depends on ← penzai.core
– models ← built on ← pz.nn + pz.core
– toolshed ← imports ← core + nn + models
– treescope ← imports ← core; hooks into IPython
– user notebooks / REPL ← imports ← all top-level APIs (pz.nn, pz.select, pz.nx, pz.ts)
– deprecated/experimental ← parallel paths ← core + nn APIs (optional)
Architectural patterns & design principles
• Modular package layering: core → nn → models → toolshed
• Functional pytree paradigm: data structures + pure functions + mutable leaf variables
• Declarative combinator pattern for building neural-network graphs
• Extension points: selectors/named-axes for custom tree rewrites; toolshed scripts for one-off model surgery; experimental subpackages for future features