# aws-neuron/aws-neuron-sdk architecture

> Architecture diagram of the GitHub repository aws-neuron/aws-neuron-sdk, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aws-neuron/aws-neuron-sdk
- Repository: https://github.com/aws-neuron/aws-neuron-sdk

## Overview

##### Project Type and Purpose
- Type: Open‐source developer toolkit / SDK for ML acceleration
- Purpose: Compile deep learning models, run them on AWS inference/training hardware (Inferentia/Trainium), and provide profiling/debugging support.

##### Components (top‐level directories & README)
- compiler/ & compiler/neuronx-cc – Model compiler CLIs and libraries
- neuron-runtime/ – Runtime driver & configuration for interacting with Neuron hardware
- neuron-customops/ – Extension points for custom C++ ops
- libnrt/ – C headers for runtime library (libnrt)
- src/ (neuronperf, neuron-gatherinfo, tools, examples, k8 configs) – Python utilities, benchmarks, sample code, Kubernetes manifests
- frameworks/ – Integration guides and API docs for PyTorch, TensorFlow, MXNet, JAX
- libraries/ – Higher‐level libraries (neuronx-distributed, nxd-inference, nxd-training, transformers-neuronx)
- containers/ – Devflow & Docker/K8 examples
- _ext/, _templates/, _static/ – Sphinx extensions, HTML templates, static assets for documentation site

##### README for Insights
- “Neuron includes a compiler, runtime driver, as well as debug and profiling utilities with a TensorBoard plugin”
- Pre‐integrated into major DL frameworks
- Provides CLI, Python APIs, containerized flows

##### Main Components
1. Developer Environment
– Python scripts / notebooks / CLI
– User’s ML framework (PyTorch / TF / MXNet / JAX)
2. Neuron SDK Front End
– CLI tools (neuron-cc, neuronx-cc)
– Python APIs (neuronperf, neuron-gatherinfo, customops interfaces)
– Sphinx‐based docs builder (for completeness, optional)
3. Compiler Subsystem
– Front-end parsing (ONNX / framework IR)
– Graph partitioner & optimizer
– Code generator targeting Neuron devices
– Output: compiled model artifacts (.neff, .so, etc.)
4. Runtime Subsystem
– libnrt driver (C library)
– Neuron-runtime daemon / kernel driver
– Configuration & tuning (nrt-configurable-parameters)
5. Hardware
– AWS EC2 instances with Inferentia / Trainium chips
– Neuron cores & interconnect
6. Profiling & Debug
– TensorBoard plugin (tensorboard-neuron, tensorboard-neuronx)
– “neuron-monitor”, “neuron-profiler” tools
7. Container & Orchestration
– Docker images (containers/docker-example)
– Kubernetes device plugin & scheduler (src/k8, containers tutorial)
8. Documentation Site
– Sphinx + custom extensions (_ext/, _templates/)
– Publishes to readthedocs via .readthedocs.yml

##### Relationships & Data Flow
- Developer → (1) invokes CLI/Python API → Compiler
- Compiler → generates Model Artifact → Runtime
- Runtime → loads Model → Dispatches operations to Hardware
- Hardware → processes tensors → returns outputs → Runtime
- Runtime → emits profiling data → Profiling Tools & TensorBoard plugin
- Developer → views profiling in TensorBoard
- Developer/CI → builds Docker images / deploys to K8 using container flows
- Documentation build (Sphinx) runs offline, publishes user guides

##### Architectural Patterns & Design Principles
- Modular, pluggable compiler & runtime
- Clear separation: front end (CLI/API), compiler core, runtime, hardware
- Extensibility: customops plugin interfaces, multiple frameworks support
- Deployment: containerized workflows, Kubernetes device plugin/scheduler
- Observability: profiling data piped into TensorBoard

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Developer Environment"
        Dev["Developer CLI / Python / Notebooks"]:::developer
    end

    subgraph "Framework Integrations"
        FW1["PyTorch"]:::developer
        FW2["TensorFlow"]:::developer
        FW3["MXNet"]:::developer
        FW4["JAX"]:::developer
    end

    subgraph "Neuron SDK Front End"
        CLI1["neuron-cc (CLI)"]:::developer
        CLI2["neuronx-cc (CLI)"]:::developer
        PYAPI1["neuronperf (Python API)"]:::developer
        PYAPI2["neuron-gatherinfo (Python API)"]:::developer
        COps["Custom Ops Extension"]:::developer
        HLL["Higher-level Libraries"]:::developer
    end

    subgraph "Compiler Subsystem"
        Parser["Front-end Parser (ONNX/IR)"]:::compile
        Optimizer["Graph Partitioner & Optimizer"]:::compile
        CodeGen["Code Generator"]:::compile
        Artf["Compiled Model Artifact (.neff/.so)"]:::compile
    end

    subgraph "Runtime Subsystem"
        LibNrt["libnrt C Library Headers"]:::runtime
        RTDaemon["Neuron-runtime Daemon / Kernel Driver"]:::runtime
        Config["Runtime Configuration & Tuning"]:::runtime
        ModelLoader["Model Loader"]:::runtime
    end

    subgraph "AWS Hardware"
        EC2["EC2 w/Inferentia & Trainium"]:::hardware
        Cores["Neuron Cores & Interconnect"]:::hardware
    end

    subgraph "Profiling & Debug"
        TBPlugin["TensorBoard Plugin"]:::profiling
        Monitor["neuron-monitor / neuron-profiler"]:::profiling
        Logs["Profiling Data Logs"]:::profiling
    end

    subgraph "Container & Orchestration"
        Docker["Docker Images & Devflows"]:::deploy
        K8s["Kubernetes Device Plugin & Scheduler"]:::deploy
    end

    subgraph "Documentation"
        SphinxExt["Sphinx Extensions (_ext)"]:::docs
        Templates["HTML Templates & Static Assets (_templates/_static)"]:::docs
        RTD[".readthedocs.yml"]:::docs
    end

    Dev -->|"uses"| CLI1
    Dev -->|"uses"| CLI2
    Dev -->|"imports"| PYAPI1
    Dev -->|"imports"| PYAPI2
    Dev -->|"builds"| COps
    Dev -->|"uses"| HLL

    FW1 --> CLI1
    FW2 --> CLI1
    FW3 --> CLI1
    FW4 --> CLI1

    CLI1 --> Parser
    CLI2 --> Parser
    PYAPI1 --> Parser
    PYAPI2 --> Parser

    Parser --> Optimizer
    Optimizer --> CodeGen
    CodeGen --> Artf

    Artf --> ModelLoader
    ModelLoader --> RTDaemon
    LibNrt --> RTDaemon
    Config --> RTDaemon
    RTDaemon --> Cores
    Cores --> RTDaemon
    RTDaemon --> ModelLoader

    RTDaemon -->|"emits"| Logs
    Logs --> Monitor
    Logs --> TBPlugin
    Monitor --> TBPlugin
    TBPlugin --> Dev

    Dev --> Docker
    Dev --> K8s
    Docker --> EC2
    K8s --> EC2

    SphinxExt --> Templates
    Templates --> RTD

    classDef developer fill:#b7e4c7,stroke:#2d6a4f,color:#000
    classDef compile fill:#caf0f8,stroke:#0077b6,color:#000
    classDef runtime fill:#ffd8a8,stroke:#fb5607,color:#000
    classDef hardware fill:#e0e0e0,stroke:#6c757d,color:#000
    classDef profiling fill:#e0bbe4,stroke:#9d4edd,color:#000
    classDef deploy fill:#d0f4de,stroke:#40916c,color:#000
    classDef docs fill:#ffcad4,stroke:#ff006e,color:#000

    click CLI1 "https://github.com/aws-neuron/aws-neuron-sdk/tree/master/compiler/neuron-cc/"
    click CLI2 "https://github.com/aws-neuron/aws-neuron-sdk/tree/master/compiler/neuronx-cc/"
    click LibNrt "https://github.com/aws-neuron/aws-neuron-sdk/tree/master/src/libnrt/include/"
    click RTDaemon "https://github.com/aws-neuron/aws-neuron-sdk/tree/master/neuron-runtime/"
    click COps "https://github.com/aws-neuron/aws-neuron-sdk/tree/master/neuron-customops/"
    click PYAPI1 "https://github.com/aws-neuron/aws-neuron-sdk/tree/master/src/neuronperf/"
    click PYAPI2 "https://github.com/aws-neuron/aws-neuron-sdk/tree/master/src/neuron-gatherinfo/"
    click Docker "https://github.com/aws-neuron/aws-neuron-sdk/tree/master/containers/"
    click K8s "https://github.com/aws-neuron/aws-neuron-sdk/tree/master/src/k8/"
    click FW1 "https://github.com/aws-neuron/aws-neuron-sdk/tree/master/frameworks/"
    click FW2 "https://github.com/aws-neuron/aws-neuron-sdk/tree/master/frameworks/"
    click FW3 "https://github.com/aws-neuron/aws-neuron-sdk/tree/master/frameworks/"
    click FW4 "https://github.com/aws-neuron/aws-neuron-sdk/tree/master/frameworks/"
    click HLL "https://github.com/aws-neuron/aws-neuron-sdk/tree/master/libraries/"
    click TBPlugin "https://github.com/aws-neuron/aws-neuron-sdk/tree/master/tools/tensorboard/"
    click Monitor "https://github.com/aws-neuron/aws-neuron-sdk/tree/master/tools/neuron-sys-tools/"
    click SphinxExt "https://github.com/aws-neuron/aws-neuron-sdk/tree/master/_ext/"
    click Templates "https://github.com/aws-neuron/aws-neuron-sdk/tree/master/_templates/"
    click Templates "https://github.com/aws-neuron/aws-neuron-sdk/tree/master/_static/"
    click RTD "https://github.com/aws-neuron/aws-neuron-sdk/blob/master/.readthedocs.yml"
```
