# ndif-team/nnsight architecture

> Architecture diagram of the GitHub repository ndif-team/nnsight, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ndif-team/nnsight
- Repository: https://github.com/ndif-team/nnsight

## Overview

##### Project type and purpose
- nnsight is an open-source Python library (“tool”) for instrumenting, tracing, and intervening in deep-learning model internals (e.g., HuggingFace Transformers, vLLM).
- It provides a high-level API (LanguageModel, .trace, .generate) that builds an “intervention graph” of proxy operations, then executes them on backends (PyTorch, vLLM, remote).
- Key use cases: hidden-state inspection, tensor-level editing, cross-prompt interventions, custom module injections.

##### File structure
– Top-level:
- src/nnsight: core library
- docs/: Sphinx-based documentation site
- tests/: pytest suite
- .github/workflows: CI/CD (lint, test, build, publish)
- pyproject.toml, pytest.ini: build config, test config
– src/nnsight subpackages:
- intervention: builds proxy graph (contexts, backends, protocols, graph/model of operations)
- tracing: parallel tracing infrastructure for control‐flow protocols
- modeling: wrappers over model‐runners (LanguageModel), vLLM executors, GPU/Ray executors
- schema: pydantic schemas for requests/responses (config, serialization)
- util, logger, config.yaml: support functions, logging, default settings
– CI/CD:
- python-app.yml: test matrix, style checks
- publish.yml: package build and PyPI upload
– Docs build:
- docs/source: .rst tutorials, Sphinx config
- docs/Makefile / make.bat: build commands

##### README for architecture insights
- The API is centered on a LanguageModel object with .trace/.generate context managers.
- Trace block builds a computation graph of “proxy” operations (save, .next(), arithmetic, assignments).
- After exiting the block, the graph is materialized: actual model load → run → return saved tensors.
- Modeling subpackage handles multiple execution backends (PyTorch, vLLM, Ray).
- Intervention subpackage orchestrates contexts, protocols (entrypoint, swap, parameter, grad, etc.), graph representation.
- Tracing subpackage handles different contexts (iterator, globals, conditional) and graph building for control flow.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Layer
    User["User Script / Notebook"]:::external

    %% API Layer
    subgraph "API Layer"
        direction TB
        LM["LanguageModel"]:::internal
        TM["Trace/Generate Managers"]:::internal
    end

    %% Intervention Engine
    subgraph "Intervention Engine"
        direction TB
        Ctxs["Contexts"]:::internal
        Protocols["Protocol Handlers"]:::internal
        subgraph "Graph Builder"
            direction TB
            GraphCore["graph.py"]:::internal
            Node["node.py"]:::internal
            Proxy["proxy.py"]:::internal
        end
        subgraph "Base & Orchestration"
            direction TB
            BaseCore["base.py"]:::internal
            Interleaver["interleaver.py"]:::internal
            Envoy["envoy.py"]:::internal
        end
    end

    %% Execution Layer
    subgraph "Execution Layer"
        direction TB
        subgraph "vLLM Backends"
            direction TB
            GPUExec["GPUExecutor"]:::internal
            RayExec["RayGPUExecutor"]:::internal
            GPURunner["GPUModelRunner"]:::internal
            Adapter["vLLM Adapter"]:::internal
        end
        RemoteExec["Remote Executor"]:::internal
    end

    %% Data Layer
    subgraph "Data Layer"
        direction TB
        ConfigSchema["config.py"]:::internal
        ReqSchema["request.py"]:::internal
        ResSchema["response.py"]:::internal
        ResultSchema["result.py"]:::internal
        FormatHelpers["format/"]:::internal
    end

    %% Support Utilities & Logging
    subgraph "Support Utilities"
        direction TB
        Util["util.py"]:::internal
        Logger["logger.py"]:::internal
        Defaults["config.yaml"]:::internal
    end

    %% External Services
    subgraph "External Services"
        direction TB
        HF["HuggingFace Hub"]:::external
        RayCluster["Ray Cluster"]:::optional
        GPUDrivers["CUDA/GPU Drivers"]:::external
    end

    %% Connections
    User -->|"import & .trace()"| LM
    LM -->|"uses methods"| TM
    TM -->|"build Graph"| GraphCore
    GraphCore -->|"defines nodes/proxies"| Adapter
    Adapter -->|"execute graph"| GPUExec
    Adapter -->|"execute graph"| RayExec
    Adapter -->|"execute graph"| RemoteExec
    GPUExec -->|"capture tensors"| ReqSchema
    RayExec -->|"capture tensors"| ReqSchema
    RemoteExec -->|"capture tensors"| ReqSchema
    GPUExec -->|"fetch weights"| HF
    GPUExec -->|"use GPU"| GPUDrivers
    RayExec -->|"use Ray"| RayCluster
    FormatHelpers -->|"serialize/format"| ResSchema
    ResSchema -->|"return results"| User

    %% Cross-layer utilities
    Util --> LM
    Util --> GraphCore
    Logger --> LM
    Logger --> BaseCore
    Defaults --> Ctxs

    %% Click Events
    click LM "https://github.com/ndif-team/nnsight/blob/main/src/nnsight/modeling/language.py"
    click TM "https://github.com/ndif-team/nnsight/blob/main/src/nnsight/tracing/contexts/tracer.py"
    click Ctxs "https://github.com/ndif-team/nnsight/tree/main/src/nnsight/intervention/contexts/"
    click Protocols "https://github.com/ndif-team/nnsight/tree/main/src/nnsight/intervention/protocols/"
    click GraphCore "https://github.com/ndif-team/nnsight/blob/main/src/nnsight/intervention/graph/graph.py"
    click Node "https://github.com/ndif-team/nnsight/blob/main/src/nnsight/intervention/graph/node.py"
    click Proxy "https://github.com/ndif-team/nnsight/blob/main/src/nnsight/intervention/graph/proxy.py"
    click BaseCore "https://github.com/ndif-team/nnsight/blob/main/src/nnsight/intervention/base.py"
    click Interleaver "https://github.com/ndif-team/nnsight/blob/main/src/nnsight/intervention/interleaver.py"
    click Envoy "https://github.com/ndif-team/nnsight/blob/main/src/nnsight/intervention/envoy.py"
    click GPUExec "https://github.com/ndif-team/nnsight/blob/main/src/nnsight/modeling/vllm/executors/GPUExecutor.py"
    click RayExec "https://github.com/ndif-team/nnsight/blob/main/src/nnsight/modeling/vllm/executors/RayGPUExecutor.py"
    click GPURunner "https://github.com/ndif-team/nnsight/blob/main/src/nnsight/modeling/vllm/model_runners/GPUModelRunner.py"
    click Adapter "https://github.com/ndif-team/nnsight/blob/main/src/nnsight/modeling/vllm/vllm.py"
    click RemoteExec "https://github.com/ndif-team/nnsight/blob/main/src/nnsight/intervention/backends/remote.py"
    click ConfigSchema "https://github.com/ndif-team/nnsight/blob/main/src/nnsight/schema/config.py"
    click ReqSchema "https://github.com/ndif-team/nnsight/blob/main/src/nnsight/schema/request.py"
    click ResSchema "https://github.com/ndif-team/nnsight/blob/main/src/nnsight/schema/response.py"
    click ResultSchema "https://github.com/ndif-team/nnsight/blob/main/src/nnsight/schema/result.py"
    click FormatHelpers "https://github.com/ndif-team/nnsight/tree/main/src/nnsight/schema/format/"
    click Util "https://github.com/ndif-team/nnsight/blob/main/src/nnsight/util.py"
    click Logger "https://github.com/ndif-team/nnsight/blob/main/src/nnsight/logger.py"
    click Defaults "https://github.com/ndif-team/nnsight/blob/main/src/nnsight/config.yaml"

    %% Styles
    classDef internal fill:#bbdefb,stroke:#1e88e5;
    classDef external fill:#c8e6c9,stroke:#388e3c;
    classDef optional fill:#fff3e0,stroke:#ffa726,stroke-dasharray: 5 5;
```
