How ndif-team/nnsight works
Project type and purpose
• nnsight is an open-source Python library (“tool”) for instrumenting, tracing, and intervening in deep-learning model internals (e.g., HuggingFace Transformers, vLLM).
• It provides a high-level API (LanguageModel, .trace, .generate) that builds an “intervention graph” of proxy operations, then executes them on backends (PyTorch, vLLM, remote).
• Key use cases: hidden-state inspection, tensor-level editing, cross-prompt interventions, custom module injections.
File structure
– Top-level:
• src/nnsight: core library
• docs/: Sphinx-based documentation site
• tests/: pytest suite
• .github/workflows: CI/CD (lint, test, build, publish)
• pyproject.toml, pytest.ini: build config, test config
– src/nnsight subpackages:
• intervention: builds proxy graph (contexts, backends, protocols, graph/model of operations)
• tracing: parallel tracing infrastructure for control‐flow protocols
• modeling: wrappers over model‐runners (LanguageModel), vLLM executors, GPU/Ray executors
• schema: pydantic schemas for requests/responses (config, serialization)
• util, logger, config.yaml: support functions, logging, default settings
– CI/CD:
• python-app.yml: test matrix, style checks
• publish.yml: package build and PyPI upload
– Docs build:
• docs/source: .rst tutorials, Sphinx config
• docs/Makefile / make.bat: build commands
README for architecture insights
• The API is centered on a LanguageModel object with .trace/.generate context managers.
• Trace block builds a computation graph of “proxy” operations (save, .next(), arithmetic, assignments).
• After exiting the block, the graph is materialized: actual model load → run → return saved tensors.
• Modeling subpackage handles multiple execution backends (PyTorch, vLLM, Ray).
• Intervention subpackage orchestrates contexts, protocols (entrypoint, swap, parameter, grad, etc.), graph representation.
• Tracing subpackage handles different contexts (iterator, globals, conditional) and graph building for control flow.