How goombalab/hnet works
Project Type and Purpose
• H-Net is an open-source deep-learning library and model implementation in PyTorch.
• Purpose: implement and train/infer the “Dynamic Chunking for End-to-End Hierarchical Sequence Modeling” (H-Net) language model, with an inference script (generate.py) and training utilities (hnet/utils/train.py).
File Structure
• Top-level components:
• configs/ (JSON model configurations)
• hnet/ (Python package containing model code and utilities)
• generate.py (inference entrypoint)
• pyproject.toml (build/install metadata)
• Inside hnet/:
• models/ (model definitions and wrappers)
• modules/ (building-block implementations: dynamic chunking, attention, MLP, rotary embeddings, etc.)
• utils/ (tokenizers, training loop)
• External dependencies: PyTorch, optional mamba_ssm (state-space model).
README Insights
• The architecture image (assets/arch.png) describes the hierarchical chunking flow.
• README emphasizes two main model classes: HNet (sequence model) and MixerSeq (language-model wrapper).
• Inference via generate.py reads configs, loads checkpoint, runs text generation.
Main Components
a. Config Layer
• Config JSON files (configs/hnet_*.json)
• Loaded by both training and inference scripts
b. Entrypoints
• Training: hnet/utils/train.py
• Inference: generate.py
c. Core Model
• hnet/models/hnet.py (HNet: (B,L,D)→(B,L,D))
• hnet/models/mixer_seq.py (wraps HNet into autoregressive LM)
d. Model Modules (building blocks)
• Dynamic Chunking (hnet/modules/dc.py)
• Isotropic blocks (hnet/modules/isotropic.py)
• Multi-head attention (mha.py), MLP (mlp.py), rotary embeddings (rotary.py), utilities (utils.py), residual blocks (block.py)
e. Utilities
• Tokenization (hnet/utils/tokenizers.py)
• Training loop, loss, optimizer setup (hnet/utils/train.py)
f. External Dependencies & Data
• PyTorch backend
• Optional mamba_ssm for state-space kernels
• Datasets on HuggingFace (FineWeb-Edu, Pile)
• Model Hub (pretrained checkpoints)
Relationships and Data Flow
1. Config JSON → parsed by Entrypoints
2. Entrypoints instantiate Core Model (via config_hnet.py)
3. Training pathway:
• Tokenizer → batches → Core Model → loss → optimizer → checkpoint
4. Inference pathway:
• generate.py loads checkpoint + config → MixerSeq → autoregressive sampling → output text
5. Core Model calls Modules: at each forward pass, dynamic chunking logic selects sequence chunks, isotropic and attention blocks process them.
Architectural Patterns & Principles
• Modular design: separate “modules” (atomic layers) from “models” (composition)
• Configuration-driven instantiation (factory pattern via config_hnet.py)
• Clear separation of training vs. inference entrypoints
• Plugin-style external dependency (mamba_ssm)
• Hierarchical sequence modeling pattern (nested chunking)