How tensorbit-labs/tensorbit-core works
1. Overall shape
Tensorbit Core is a single C++20 codebase with an optional CUDA acceleration layer. Its main job is to ingest model weights, compute pruning importance scores, apply structured sparsity, and serialize the result into a compact .tb artifact for downstream stages.
2. Primary pipeline
The central data flow is:
.safetensors model weights → loader/parse → EHAP scoring → CORING mask selection/pruning → .tb serialization
This is the core architecture to reflect in a diagram; everything else supports one of these stages.
3. Loader boundary
include/tensorbit/core/loader.hpp and related code provide the model ingestion boundary. The README says it supports header-only parsing of HuggingFace safetensors weights, including F32/F16/BF16/I64. This is the input interface between external model files and Tensorbit’s internal tensor representation.
4. Tensor and common core types
tensor.hpp, common.hpp, and serialization.hpp define the shared in-memory abstractions and utility types used across the pipeline. These are the “internal contract” layer between parsing, scoring, pruning, and output. The repo emphasizes modern C++ utilities such as Result<T,E>, spans, and concepts.
5. EHAP subsystem: importance estimation
ehap.hpp / src/ehap.cpp implement the Hessian-aware pruning logic. This is the analytical stage that estimates parameter importance using second-order information, Fisher/EMA-style approximation, and OBS/OBD-style scoring. Architecturally, it consumes tensors and gradient/statistical state, then emits importance scores or pruning decisions.
6. CORING subsystem: structured sparsity enforcement
coring.hpp / src/coring.cpp implement N:M structured sparsity selection and refinement. This is the masking/pruning enforcement boundary: it converts importance information into a hardware-friendly sparse pattern, especially 2:4. It is the stage that transforms “which weights matter” into “which weights remain.”
7. CUDA execution layer
kernels.hpp, src/kernels.cu, and src/kernels_stubs.cpp form the acceleration boundary. The codebase supports GPU-optimized kernels when CUDA is enabled, with CPU-only fallback via stub implementations. For architecture diagrams, treat this as an optional compute backend under the pruning/masking stages, not as a separate product surface.
8. Serialization/output boundary
src/serialization.cpp and include/tensorbit/core/serialization.hpp convert the pruned model into the .tb format. The README highlights a fixed 4096-byte header, packed N:M bitmasks, and FP32/FP16/BF16 storage. This is the persistence boundary between Tensorbit Core and later pipeline stages.
9. CLI / orchestration entry point
src/main.cpp is the executable entry point for the pruning tool. It likely coordinates model loading, optional mock/demo sizing, algorithm selection, pruning execution, and output writing. In the architecture diagram, this is the top-level control plane that sequences the pipeline stages.
10. Build and platform model
The project uses CMake and is designed for portable C++20 builds with optional CUDA 12 support. The dependency footprint is intentionally small: Eigen3 on the CPU side and CUDA on supported systems. That makes the repository a self-contained engine rather than a distributed service.
11. Testing structure
Tests are split by subsystem: tests/test_coring.cpp and tests/test_ehap.cpp cover the algorithmic cores, while tests/merge and tests/multi_tensor exercise serialization/model-composition workflows via scripts. This suggests the repo validates both mathematical correctness and end-to-end format behavior.
12. Utility scripts and operational aids
scripts/download_model.py, fix_shapes.py, merge_tbm.py, and setup scripts support data preparation and environment setup rather than the core runtime. They sit around the engine to make local and cloud workflows practical, especially for large-model experiments.
13. Documentation as architecture source
docs/ARCHITECTURE.md, docs/ALGORITHMS.md, docs/EHAP.md, and docs/CORING.md indicate that the repo is split into clear mathematical, format, and systems concerns. The documentation mirrors the code boundaries and is a good source for naming the diagram’s main subsystems.
14. What to draw as the major boundaries
The most important architectural boundaries are:
- External model format ingestion vs internal tensor representation
- EHAP scoring vs CORING mask selection
- CPU fallback vs CUDA accelerated kernels
- internal pruned model state vs serialized .tb artifact
- core library code vs test and utility tooling
Main entry points: coring.cpp CORING impl · ehap.hpp EHAP · kernels.hpp Kernels · loader.hpp Loader
Components 17
Core pipeline
Primary model ingestion, scoring, pruning, and serialization flow.
- CLI · Top-level pruning tool that orchestrates the pipeline.
src/main.cpp - Loader · Boundary for parsing safetensors model weights into internal tensors.
include/tensorbit/core/loader.hpp - Tensor core · Shared tensor and utility abstractions used across the pipeline.
include/tensorbit/core/tensor.hpp - Common types · Result, span, concept, and shared helper definitions.
include/tensorbit/core/common.hpp - EHAP · Hessian-aware importance estimation and pruning score computation.
include/tensorbit/core/ehap.hpp - EHAP impl · Implementation of second-order scoring and statistical approximation.
src/ehap.cpp - CORING · Structured sparsity selection and refinement boundary.
include/tensorbit/core/coring.hpp - CORING impl · Implementation of N:M mask selection and pruning refinement.
src/coring.cpp - Serializer · Converts pruned model state into compact .tb artifacts.
include/tensorbit/core/serialization.hpp - TB writer · Implementation of header packing, mask encoding, and weight emission.
src/serialization.cpp
Execution backend
Optional compute acceleration layer for pruning kernels.
- Kernels · Acceleration boundary for compute kernels used by pruning stages.
include/tensorbit/core/kernels.hpp - CUDA kernels · GPU implementation of accelerated compute paths.
src/kernels.cu - CPU stubs · CPU fallback implementations when CUDA is unavailable.
src/kernels_stubs.cpp
Tooling and validation
Supporting scripts, tests, and documentation around the engine.
- EHAP tests · Unit coverage for importance estimation logic.
tests/test_ehap.cpp - CORING tests · Unit coverage for structured sparsity and pruning selection.
tests/test_coring.cpp - Scripts · Workflow helpers for model download, shape fixing, merge, and setup.
scripts/download_model.py - Docs · Architecture and algorithm documentation that mirrors the code boundaries.
docs/ARCHITECTURE.md
Connections 18
- CLI → Loader: loads · CLI starts by ingesting source model weights.
- Loader → Tensor core: produces · Parsed weights become internal tensor objects.
- Tensor core → EHAP: feeds · Tensor state is consumed by importance estimation.
- Common types → EHAP: supports · Shared contracts underpin the analysis stage.
- EHAP → EHAP impl: implemented by · Public scoring boundary maps to the EHAP engine.
- EHAP impl → CORING: scores · Importance estimates drive sparsity selection.
- CORING → CORING impl: implemented by · Public masking boundary maps to CORING logic.
- CORING impl → Kernels: accelerates via · Pruning can dispatch into backend kernels.
- Kernels → CUDA kernels: gpu path · CUDA backend is used when enabled.
- Kernels → CPU stubs: fallback · CPU stubs provide the non-CUDA path.
- CORING impl → Serializer: outputs · Pruned weights are prepared for serialization.
- Serializer → TB writer: implemented by · Output boundary maps to TB format writer.
- TB writer → CLI: writes · CLI completes the pipeline by emitting the artifact.
- EHAP tests → EHAP impl: validates · EHAP tests check analytical scoring behavior.
- CORING tests → CORING impl: validates · CORING tests check structured sparsity behavior.
- Scripts → Loader: prepares input · Scripts help obtain and reshape source models.
- Docs → EHAP: describes · Documentation captures the analytical subsystem.
- Docs → CORING: describes · Documentation captures the sparsity subsystem.