# google-deepmind/penzai architecture

> Architecture diagram of the GitHub repository google-deepmind/penzai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/google-deepmind/penzai
- Repository: https://github.com/google-deepmind/penzai

## Overview

##### Project type and purpose
- Type: Python library (open-source toolkit) built on JAX.
- Purpose:
– Represent neural networks as functional pytree data structures.
– Expose a declarative combinator-based NN API, a pretty-printer (Treescope), and model-surgery “toolshed” scripts.

##### File structure
- Top-level directories:
– penzai/            ← the main source package
– docs/              ← Sphinx documentation sources
– notebooks/         ← tutorial and demo notebooks
– tests/             ← unit and integration tests
– .github/, CI files ← CI pipelines (unittests.yml)
– pyproject.toml     ← packaging/build config
- Inside penzai/: logical subpackages:
– core/              ← tree and array utilities (selectors, named_axes, partitioning, random_stream, struct, shapecheck)
– nn/                ← declarative neural-network library (layers, combinators, parameters, layer_stack)
– models/            ← reference model definitions (simple_mlp; transformer variants: Gemma, GPT-NeoX, Llama, Mistral)
– pz/                ← alias namespace (pz.nn, pz.ts)
– toolshed/          ← utility scripts for training, checkpointing, visualization, model re-wiring
– treescope/         ← interactive pretty-printing (plugin for IPython/Colab)
– deprecated/v1/     ← legacy API for backwards compatibility
– experimental/v2/   ← early-stage API prototypes

##### README for additional insights
- Four “modules” highlighted:
1. Treescope (`penzai.treescope`, `pz.ts`)
2. Core utilities (`penzai.core`, `pz.select`, `pz.nx`)
3. Declarative NN (`penzai.nn`, `pz.nn`)
4. Transformer models (`penzai.models.transformer`)
- Integration points:
– User code (Jupyter/IPython) ↔ Treescope for visualization
– pz.select and pz.nx ↔ tree-traversal and named-axis APIs
– pz.nn layers ↔ core utilities for parameter management and stateful leaves
– Models built on top of pz.nn + pz.nx + pz.select

##### Main components & their relationships
A. External Dependencies (bottom layer)
– JAX (jax.numpy, jax.random, jax.tree_util)
– IPython/Colab display system
B. Core Package (`penzai.core`)
– Selectors module (pz.select)
– Named-axes module (pz.nx)
– Tree utilities, partitioning, random_stream, struct, shapecheck
C. Declarative Neural Net Library (`penzai.nn`, alias `pz.nn`)
– Layer base classes, Elementwise, combinators, linear/affine, dropout, embeddings, grouping, layer_stack
– Uses core utilities for mutable state, parameter sharing
D. Transformer & Model Package (`penzai.models.transformer`, `penzai.models.simple_mlp`)
– Model parts, sampling loops, pre-trained weights loading
– Uses pz.nn + pz.nx + pz.select
E. Treescope Visualization (`penzai.treescope`, alias `pz.ts`)
– Hooks into IPython/Colab display
– Renders arbitrary JAX pytrees and NDArrays
F. Toolshed Scripts (`penzai.toolshed`)
– Training helpers, gradient checkpointing, model rewiring, unflaxify, etc.
– Operate as standalone scripts or importable utilities
G. Deprecated & Experimental APIs
H. Documentation & Notebooks (side panel)
– docs/ ← Sphinx
– notebooks/ ← interactive demos

– Core ← calls/invokes ← External JAX APIs
– pz.nn ← depends on ← penzai.core
– models ← built on ← pz.nn + pz.core
– toolshed ← imports ← core + nn + models
– treescope ← imports ← core; hooks into IPython
– user notebooks / REPL ← imports ← all top-level APIs (pz.nn, pz.select, pz.nx, pz.ts)
– deprecated/experimental ← parallel paths ← core + nn APIs (optional)

##### Architectural patterns & design principles
- Modular package layering: core → nn → models → toolshed
- Functional pytree paradigm: data structures + pure functions + mutable leaf variables
- Declarative combinator pattern for building neural-network graphs
- Extension points: selectors/named-axes for custom tree rewrites; toolshed scripts for one-off model surgery; experimental subpackages for future features

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Dependencies
    subgraph "External Dependencies"
        JAX(("JAX (jax.numpy, jax.random, jax.tree_util)")):::external
        IPY(("IPython/Colab Display")):::external
    end

    %% Core Package
    subgraph "Core Utilities" 
        direction TB
        SELECTORS["selectors.py"]:::core
        NAMED_AXES["named_axes.py"]:::core
        PARTITIONING["partitioning.py"]:::core
        RANDOM_STREAM["random_stream.py"]:::core
        STRUCT["struct.py"]:::core
        SHAPECHECK["shapecheck.py"]:::core
        TREE_UTIL["tree_util.py"]:::core
        VARIABLES["variables.py"]:::core
    end

    %% Neural Network Library
    subgraph "Declarative NN Library" 
        direction TB
        LAYER["layer.py"]:::nn
        LINEAR_AFFINE["linear_and_affine.py"]:::nn
        ATTENTION["attention.py"]:::nn
        DROPOUT["dropout.py"]:::nn
        EMBEDDINGS["embeddings.py"]:::nn
        COMBINATORS["combinators.py"]:::nn
        GROUPING["grouping.py"]:::nn
        PARAMETERS["parameters.py"]:::nn
        LAYER_STACK["layer_stack.py"]:::nn
    end

    %% Models Package
    subgraph "Models Package"
        direction TB
        SIMPLE_MLP["simple_mlp.py"]:::models
        subgraph "Transformer" 
            direction TB
            MODEL_PARTS["model_parts.py"]:::models
            SAMPLING_MODE["sampling_mode.py"]:::models
            DECODING_LOOP["simple_decoding_loop.py"]:::models
            VARIANTS["variants/"]:::models
        end
    end

    %% Treescope Visualization
    subgraph "Treescope Visualization"
        direction TB
        FORMAT_UTIL["formatting_util.py"]:::viz
        REPR_LIB["repr_lib.py"]:::viz
        COMPAT_SETUP["_compatibility_setup.py"]:::viz
    end

    %% Toolshed Scripts
    subgraph "Toolshed Scripts"
        direction TB
        BASIC_TRAIN["basic_training.py"]:::toolshed
        GRAD_CHECK["gradient_checkpointing.py"]:::toolshed
        MODEL_REWIRE["model_rewiring.py"]:::toolshed
        UNFLAXIFY["unflaxify.py"]:::toolshed
        ISOLATE_SUB["isolate_submodel.py"]:::toolshed
        JIT_WRAP["jit_wrapper.py"]:::toolshed
        LORA["lora.py"]:::toolshed
    end

    %% Alias Namespace
    subgraph "Alias Namespace"
        direction TB
        PZ_NN["pz/nn.py"]:::alias
        PZ_TS["pz/ts.py"]:::alias
    end

    %% Deprecated & Experimental
    subgraph "Legacy & Experimental" 
        direction TB
        classDef dashed fill-opacity:0 stroke-dasharray: 5 5
        DEPRECATED["penzai/deprecated/v1/"]:::dashed
        EXPERIMENTAL["penzai/experimental/v2/"]:::dashed
    end

    %% Documentation & Notebooks
    DOCS[/docs/]:::docs
    NOTEBOOKS[/notebooks/]:::docs

    %% Relationships
    JAX -->|"calls/invokes"| SELECTORS
    JAX -->|"calls/invokes"| NAMED_AXES
    JAX -->|"calls/invokes"| PARTITIONING
    JAX -->|"calls/invokes"| RANDOM_STREAM
    JAX -->|"calls/invokes"| STRUCT
    JAX -->|"calls/invokes"| SHAPECHECK
    JAX -->|"calls/invokes"| TREE_UTIL
    JAX -->|"calls/invokes"| VARIABLES

    SELECTORS -->|"used by"| LAYER
    NAMED_AXES -->|"used by"| LAYER
    TREE_UTIL -->|"used by"| PARAMETERS

    LAYER -->|"depends on"| STRUCT
    LAYER -->|"depends on"| VARIABLES
    LINEAR_AFFINE -->|"combines"| LAYER
    ATTENTION -->|"combines"| LINEAR_AFFINE
    DROPOUT -->|"combines"| LAYER
    EMBEDDINGS -->|"combines"| LAYER
    COMBINATORS -->|"composes"| LAYER
    GROUPING -->|"organizes"| PARAMETERS
    LAYER_STACK -->|"stacks"| LAYER

    SIMPLE_MLP -->|"built on"| LAYER
    MODEL_PARTS -->|"built on"| ATTENTION
    SAMPLING_MODE -->|"built on"| MODEL_PARTS
    DECODING_LOOP -->|"built on"| SAMPLING_MODE
    VARIANTS -->|"extends"| MODEL_PARTS

    BASIC_TRAIN -->|"imports"| SELECTORS
    BASIC_TRAIN -->|"imports"| LAYER
    GRAD_CHECK -->|"imports"| RANDOM_STREAM
    MODEL_REWIRE -->|"imports"| TREE_UTIL
    UNFLAXIFY -->|"imports"| STRUCT
    ISOLATE_SUB -->|"imports"| PARAMETERS
    JIT_WRAP -->|"imports"| COMBINATORS
    LORA -->|"imports"| LINEAR_AFFINE

    FORMAT_UTIL -->|"imports"| TREE_UTIL
    REPR_LIB -->|"imports"| STRUCT
    COMPAT_SETUP -->|"hooks into"| IPY

    IPY -->|"hosts"| FORMAT_UTIL
    IPY -->|"hosts"| REPR_LIB

    PZ_NN -->|"alias for"| LAYER
    PZ_NN -->|"alias for"| COMBINATORS
    PZ_TS -->|"alias for"| FORMAT_UTIL
    PZ_TS -->|"alias for"| REPR_LIB

    DEPRECATED -.->|"optional"| SELECTORS
    DEPRECATED -.->|"optional"| LAYER
    EXPERIMENTAL -.->|"prototype"| SELECTORS
    EXPERIMENTAL -.->|"prototype"| LAYER

    DOCS -->|"documents"| SELECTORS
    DOCS -->|"documents"| LAYER
    NOTEBOOKS -->|"demos"| FORMAT_UTIL
    NOTEBOOKS -->|"demos"| LAYER

    %% Click Events
    click SELECTORS "https://github.com/google-deepmind/penzai/blob/main/penzai/core/selectors.py"
    click NAMED_AXES "https://github.com/google-deepmind/penzai/blob/main/penzai/core/named_axes.py"
    click PARTITIONING "https://github.com/google-deepmind/penzai/blob/main/penzai/core/partitioning.py"
    click RANDOM_STREAM "https://github.com/google-deepmind/penzai/blob/main/penzai/core/random_stream.py"
    click STRUCT "https://github.com/google-deepmind/penzai/blob/main/penzai/core/struct.py"
    click SHAPECHECK "https://github.com/google-deepmind/penzai/blob/main/penzai/core/shapecheck.py"
    click TREE_UTIL "https://github.com/google-deepmind/penzai/blob/main/penzai/core/tree_util.py"
    click VARIABLES "https://github.com/google-deepmind/penzai/blob/main/penzai/core/variables.py"

    click LAYER "https://github.com/google-deepmind/penzai/blob/main/penzai/nn/layer.py"
    click LINEAR_AFFINE "https://github.com/google-deepmind/penzai/blob/main/penzai/nn/linear_and_affine.py"
    click ATTENTION "https://github.com/google-deepmind/penzai/blob/main/penzai/nn/attention.py"
    click DROPOUT "https://github.com/google-deepmind/penzai/blob/main/penzai/nn/dropout.py"
    click EMBEDDINGS "https://github.com/google-deepmind/penzai/blob/main/penzai/nn/embeddings.py"
    click COMBINATORS "https://github.com/google-deepmind/penzai/blob/main/penzai/nn/combinators.py"
    click GROUPING "https://github.com/google-deepmind/penzai/blob/main/penzai/nn/grouping.py"
    click PARAMETERS "https://github.com/google-deepmind/penzai/blob/main/penzai/nn/parameters.py"
    click LAYER_STACK "https://github.com/google-deepmind/penzai/blob/main/penzai/nn/layer_stack.py"

    click SIMPLE_MLP "https://github.com/google-deepmind/penzai/blob/main/penzai/models/simple_mlp.py"
    click MODEL_PARTS "https://github.com/google-deepmind/penzai/blob/main/penzai/models/transformer/model_parts.py"
    click SAMPLING_MODE "https://github.com/google-deepmind/penzai/blob/main/penzai/models/transformer/sampling_mode.py"
    click DECODING_LOOP "https://github.com/google-deepmind/penzai/blob/main/penzai/models/transformer/simple_decoding_loop.py"
    click VARIANTS "https://github.com/google-deepmind/penzai/tree/main/penzai/models/transformer/variants/"

    click FORMAT_UTIL "https://github.com/google-deepmind/penzai/blob/main/penzai/treescope/formatting_util.py"
    click REPR_LIB "https://github.com/google-deepmind/penzai/blob/main/penzai/treescope/repr_lib.py"
    click COMPAT_SETUP "https://github.com/google-deepmind/penzai/blob/main/penzai/treescope/_compatibility_setup.py"

    click BASIC_TRAIN "https://github.com/google-deepmind/penzai/blob/main/penzai/toolshed/basic_training.py"
    click GRAD_CHECK "https://github.com/google-deepmind/penzai/blob/main/penzai/toolshed/gradient_checkpointing.py"
    click MODEL_REWIRE "https://github.com/google-deepmind/penzai/blob/main/penzai/toolshed/model_rewiring.py"
    click UNFLAXIFY "https://github.com/google-deepmind/penzai/blob/main/penzai/toolshed/unflaxify.py"
    click ISOLATE_SUB "https://github.com/google-deepmind/penzai/blob/main/penzai/toolshed/isolate_submodel.py"
    click JIT_WRAP "https://github.com/google-deepmind/penzai/blob/main/penzai/toolshed/jit_wrapper.py"
    click LORA "https://github.com/google-deepmind/penzai/blob/main/penzai/toolshed/lora.py"

    click PZ_NN "https://github.com/google-deepmind/penzai/blob/main/penzai/pz/nn.py"
    click PZ_TS "https://github.com/google-deepmind/penzai/blob/main/penzai/pz/ts.py"

    click DOCS "https://github.com/google-deepmind/penzai/tree/main/docs/"
    click NOTEBOOKS "https://github.com/google-deepmind/penzai/tree/main/notebooks/"

    %% Styles
    classDef core fill:#bfdfff,stroke:#3399ff
    classDef nn fill:#bfffbf,stroke:#33aa33
    classDef models fill:#e0b3ff,stroke:#8844cc
    classDef viz fill:#ffd9b3,stroke:#ff8c00
    classDef toolshed fill:#b3f2f2,stroke:#009999
    classDef alias fill:#f2e6b3,stroke:#999900
    classDef external fill:#e0e0e0,stroke:#888888,stroke-width:1px
    classDef docs fill:#fff5b3,stroke:#e6c200,stroke-width:1px
    classDef dashed fill:#ffffff,stroke:#cccccc,stroke-dasharray:5 5,stroke-width:1px
    class SELECTORS,NAMED_AXES,PARTITIONING,RANDOM_STREAM,STRUCT,SHAPECHECK,TREE_UTIL,VARIABLES core
    class LAYER,LINEAR_AFFINE,ATTENTION,DROPOUT,EMBEDDINGS,COMBINATORS,GROUPING,PARAMETERS,LAYER_STACK nn
    class SIMPLE_MLP,MODEL_PARTS,SAMPLING_MODE,DECODING_LOOP,VARIANTS models
    class FORMAT_UTIL,REPR_LIB,COMPAT_SETUP viz
    class BASIC_TRAIN,GRAD_CHECK,MODEL_REWIRE,UNFLAXIFY,ISOLATE_SUB,JIT_WRAP,LORA toolshed
    class PZ_NN,PZ_TS alias
    class JAX,IPY external
    class DOCS,NOTEBOOKS docs
```
