# goombalab/hnet architecture

> Architecture diagram of the GitHub repository goombalab/hnet, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/goombalab/hnet
- Repository: https://github.com/goombalab/hnet

## Overview

##### Project Type and Purpose
- H-Net is an open-source deep-learning library and model implementation in PyTorch.
- Purpose: implement and train/infer the “Dynamic Chunking for End-to-End Hierarchical Sequence Modeling” (H-Net) language model, with an inference script (`generate.py`) and training utilities (`hnet/utils/train.py`).

##### File Structure
- Top-level components:
- configs/   (JSON model configurations)
- hnet/      (Python package containing model code and utilities)
- generate.py (inference entrypoint)
- pyproject.toml (build/install metadata)
- Inside `hnet/`:
- models/    (model definitions and wrappers)
- modules/   (building-block implementations: dynamic chunking, attention, MLP, rotary embeddings, etc.)
- utils/     (tokenizers, training loop)
- External dependencies: PyTorch, optional mamba_ssm (state-space model).

##### README Insights
- The architecture image (`assets/arch.png`) describes the hierarchical chunking flow.
- README emphasizes two main model classes: `HNet` (sequence model) and `MixerSeq` (language-model wrapper).
- Inference via `generate.py` reads configs, loads checkpoint, runs text generation.

##### Main Components
a. Config Layer
- Config JSON files (configs/hnet_*.json)
- Loaded by both training and inference scripts
b. Entrypoints
- Training: `hnet/utils/train.py`
- Inference: `generate.py`
c. Core Model
- hnet/models/hnet.py (HNet: (B,L,D)→(B,L,D))
- hnet/models/mixer_seq.py (wraps HNet into autoregressive LM)
d. Model Modules (building blocks)
- Dynamic Chunking (`hnet/modules/dc.py`)
- Isotropic blocks (`hnet/modules/isotropic.py`)
- Multi-head attention (`mha.py`), MLP (`mlp.py`), rotary embeddings (`rotary.py`), utilities (`utils.py`), residual blocks (`block.py`)
e. Utilities
- Tokenization (`hnet/utils/tokenizers.py`)
- Training loop, loss, optimizer setup (`hnet/utils/train.py`)
f. External Dependencies & Data
- PyTorch backend
- Optional mamba_ssm for state-space kernels
- Datasets on HuggingFace (FineWeb-Edu, Pile)
- Model Hub (pretrained checkpoints)

##### Relationships and Data Flow
1. Config JSON → parsed by Entrypoints
2. Entrypoints instantiate Core Model (via config_hnet.py)
3. Training pathway:
- Tokenizer → batches → Core Model → loss → optimizer → checkpoint
4. Inference pathway:
- generate.py loads checkpoint + config → MixerSeq → autoregressive sampling → output text
5. Core Model calls Modules: at each forward pass, dynamic chunking logic selects sequence chunks, isotropic and attention blocks process them.

##### Architectural Patterns & Principles
- Modular design: separate “modules” (atomic layers) from “models” (composition)
- Configuration-driven instantiation (factory pattern via config_hnet.py)
- Clear separation of training vs. inference entrypoints
- Plugin-style external dependency (mamba_ssm)
- Hierarchical sequence modeling pattern (nested chunking)

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Config Layer"
        Configs["Configs (configs/*.json)"]:::config
        Meta["pyproject.toml"]:::config
    end

    subgraph "Entrypoints"
        Generate["generate.py"]:::entry
        Train["train.py (hnet/utils/train.py)"]:::entry
    end

    subgraph "Core Model"
        ConfigFactory["config_hnet.py"]:::model
        HNet["HNet (hnet/models/hnet.py)"]:::model
        MixerSeq["MixerSeq (hnet/models/mixer_seq.py)"]:::model
    end

    subgraph "Modules"
        DC["DynamicChunking (hnet/modules/dc.py)"]:::module
        ISO["IsotropicBlock (hnet/modules/isotropic.py)"]:::module
        MHA["MultiHeadAttention (hnet/modules/mha.py)"]:::module
        MLP["MLPLayer (hnet/modules/mlp.py)"]:::module
        ROT["RotaryEmbeddings (hnet/modules/rotary.py)"]:::module
        BLK["ResidualBlocks (hnet/modules/block.py)"]:::module
        MODUTIL["ModuleUtils (hnet/modules/utils.py)"]:::module
    end

    subgraph "Utilities"
        Tok["Tokenizers (hnet/utils/tokenizers.py)"]:::util
    end

    subgraph "External Dependencies"
        Torch["PyTorch"]:::external
        SSM["mamba_ssm"]:::external
        HFDATA["HF Datasets"]:::external
        HFMODEL["HF Model Hub"]:::external
        Arch["arch.png (assets/arch.png)"]:::external
    end

    Output["GeneratedText"]:::output

    %% Relationships
    Configs -->|"loadconfig"| Generate
    Configs -->|"loadconfig"| Train
    Meta -->|"readmetadata"| Generate
    Meta -->|"readmetadata"| Train

    Generate -->|"instantiate"| ConfigFactory
    Train -->|"instantiate"| ConfigFactory

    ConfigFactory -->|"create"| HNet
    ConfigFactory -->|"wrap"| MixerSeq

    Train -->|"tokenize"| Tok
    Tok -->|"inputbatch"| HNet

    HNet -->|"calls"| DC
    HNet -->|"calls"| ISO
    HNet -->|"calls"| MHA
    HNet -->|"calls"| MLP
    HNet -->|"calls"| ROT
    HNet -->|"calls"| BLK
    HNet -->|"calls"| MODUTIL

    Generate -->|"call"| MixerSeq
    MixerSeq -->|"sampletokens"| Output

    Train -->|"uses"| Torch
    Generate -->|"uses"| Torch

    DC -->|"optionalplugin"| SSM

    Train -->|"loaddata"| HFDATA
    Generate -->|"loadcheckpoint"| HFMODEL

    %% Click Events
    click Configs "https://github.com/goombalab/hnet/tree/main/configs/"
    click Generate "https://github.com/goombalab/hnet/blob/main/generate.py"
    click Train "https://github.com/goombalab/hnet/blob/main/hnet/utils/train.py"
    click HNet "https://github.com/goombalab/hnet/blob/main/hnet/models/hnet.py"
    click MixerSeq "https://github.com/goombalab/hnet/blob/main/hnet/models/mixer_seq.py"
    click ConfigFactory "https://github.com/goombalab/hnet/blob/main/hnet/models/config_hnet.py"
    click DC "https://github.com/goombalab/hnet/blob/main/hnet/modules/dc.py"
    click ISO "https://github.com/goombalab/hnet/blob/main/hnet/modules/isotropic.py"
    click MHA "https://github.com/goombalab/hnet/blob/main/hnet/modules/mha.py"
    click MLP "https://github.com/goombalab/hnet/blob/main/hnet/modules/mlp.py"
    click ROT "https://github.com/goombalab/hnet/blob/main/hnet/modules/rotary.py"
    click BLK "https://github.com/goombalab/hnet/blob/main/hnet/modules/block.py"
    click MODUTIL "https://github.com/goombalab/hnet/blob/main/hnet/modules/utils.py"
    click Tok "https://github.com/goombalab/hnet/blob/main/hnet/utils/tokenizers.py"
    click Meta "https://github.com/goombalab/hnet/blob/main/pyproject.toml"
    click Arch "https://github.com/goombalab/hnet/blob/main/assets/arch.png"

    %% Styles
    classDef config fill:#ffa500,stroke:#333,stroke-width:1px
    classDef entry fill:#90ee90,stroke:#333,stroke-width:1px
    classDef model fill:#add8e6,stroke:#333,stroke-width:1px
    classDef module fill:#87cefa,stroke:#333,stroke-width:1px
    classDef util fill:#add8e6,stroke:#333,stroke-width:1px
    classDef external fill:#ffe699,stroke:#333,stroke-width:1px
    classDef output fill:#ffe0b2,stroke:#333,stroke-width:1px
```
