# arm-software/armnn architecture

> Architecture diagram of the GitHub repository arm-software/armnn, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arm-software/armnn
- Repository: https://github.com/arm-software/armnn

## Overview

##### Project Type and Purpose
- Arm NN is an open-source C++17 SDK—inference engine—designed to accelerate ML models (TF Lite, ONNX) on Arm CPUs, GPUs, NPUs.
- It provides:
- A core runtime for loading and executing networks.
- Parsers (TF Lite, ONNX) and a TF Lite Delegate (Python/C++).
- A plugin-style backend registry with CPU (reference, Neon), GPU (OpenCL via ACL), NPU (Ethos-N driver) backends.
- Profiling infrastructure (client, server, common).
- Serialization/deserialization, build tools, samples, tests, third-party libs.

##### File Structure Analysis (Top-Level)
- CMakeLists.txt, scripts, docker → build orchestration (Arm NN Build Tool).
- /, src/armnn → core runtime and network representation.
- src/parsers (armnnTfLiteParser, armnnOnnxParser), src/serializer, src/deserializer → model I/O.
- delegate/ → TF Lite Delegate (JNI, classic, opaque) and `delegate/python`.
- src/backends → plugin backends (aclCommon, cl, neon, reference, tosa… dynamic).
- profiling/ → profiling client, common, server components.
- samples/, tests/, docs/, third-party/ → examples, test harnesses, documentation, external libs.

##### README Insights
- Emphasizes: core runtime, delegate pattern for unsupported operators, parser libraries, backend plugin architecture (registry, dynamic loading), build tool for dependencies (ACL).
- Mentions Python interface for TF Lite Delegate and Debian/Android binaries.

1. Client Application
- C++ API or Python (TF Lite Delegate)
2. Delegate / Parser Layer
- TF Lite Delegate (fallback to TFLite runtime)
- TF Lite Parser, ONNX Parser
3. Arm NN Core Runtime
- Network loader, optimizer, execution frame, tensor management
4. Backend Registry & Plugin Loader
5. Compute Backends (plugin modules)
- CPU Reference, Neon
- GPU (OpenCL via ACL)
- NPU (Ethos-N driver)
- Dynamic/backendsCommon, TOSA, etc.
6. Serialization/Deserialization
- armnnSerializer, armnnDeserializer
7. Profiling Subsystem
- Profiling Client (inferred from runtime) → Profiling Connection → Profiling Server
8. Build Tool & Scripts
- Docker, bash scripts to build Arm NN + ACL + dependencies
9. Samples & Tests (execute network, delegates, backends)

- Client → Delegate/Parser (load model) → Core Runtime
- Core Runtime → Optimizer → SubgraphViewSelector → ExecutionFrame
- ExecutionFrame → Backend Registry → Selected Backend Plugin
- Backend Plugin → Hardware (CPU, GPU, NPU)
- Core Runtime ↔ Profiling Client → Profiling Connection → Profiling Server → Packet Handler
- Serialization/Deserialization components interface with Core for graph I/O.
- Build Tool → Generates libraries for core, parsers, backends, profiling.
- Samples/Tests → Link against built libraries to validate functionality.

c. Architectural Patterns and Principles
- Modular plugin architecture: backends register via registry and can be loaded dynamically.
- Delegate/Fallback pattern: TF Lite Delegate accelerates supported ops; falls back on reference runtime.
- Layered design: API → Parser/Delegate → Core Runtime → Backend → Hardware.
- Separation of concerns: profiling separated into client/common/server modules.

d. Key Technologies
- C++17, CMake, Docker, bash for build automation.
- Python bindings for TF Lite Delegate.
- Arm Compute Library (ACL) for GPU kernels.
- Ethos-N NPU driver.
- Profiling via sockets, pipe server, timeline decoder.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Application
    subgraph "Client Application"
        ClientCpp["C++ API Executables & Tests"]:::client
        PythonDel["Python Delegate Interface"]:::client
    end

    %% Delegate / Parser Layer
    subgraph "Delegate / Parser Layer"
        TFLiteDelClassic["TF-Lite Delegate (Classic)"]:::parser
        TFLiteDelOpaque["TF-Lite Delegate (Opaque)"]:::parser
        TFLiteDelJNI["TF-Lite Delegate JNI"]:::parser
        TfLiteParser["TF-Lite Parser"]:::parser
        ONNXParser["ONNX Parser"]:::parser
    end

    %% Core Runtime
    subgraph "Arm NN Core Runtime"
        CoreRuntime["Core Runtime"]:::core
    end

    %% Backend Registry & Plugin Loader
    subgraph "Backend Registry & Plugin Loader"
        BackendRegistry["Backend Registry"]:::core
    end

    %% Compute Backends
    subgraph "Compute Backends (Plugins)"
        RefBackend["Reference CPU Backend"]:::backend
        NeonBackend["NEON CPU Backend"]:::backend
        CLBackend["OpenCL GPU Backend"]:::backend
        ACLCommon["ACL Common Utilities"]:::backend
        TosaCommon["TOSA Common Utilities"]:::backend
        TosaRef["TOSA Reference Backend"]:::backend
    end

    %% Serialization / Deserialization
    subgraph "Serialization / Deserialization"
        Serializer["Serializer"]:::core
        Deserializer["Deserializer"]:::core
    end

    %% Profiling Subsystem
    subgraph "Profiling Subsystem"
        ProfClient["Profiling Client"]:::profiling
        ProfCommon["Profiling Common"]:::profiling
        ProfServer["Profiling Server"]:::profiling
    end

    %% Build Tool & Scripts
    subgraph "Build Tool & Scripts"
        Docker["Docker & Build Scripts"]:::tooling
        CMake["CMake Helpers & CMakeLists.txt"]:::tooling
        Scripts["Utility Scripts"]:::tooling
    end

    %% Samples & Tests
    subgraph "Samples & Tests"
        Samples["Example Applications"]:::tooling
        Tests["Unit & Integration Tests"]:::tooling
    end

    %% Connections: Client to Parser/Delegate
    ClientCpp -->|load model| TfLiteParser
    ClientCpp -->|load model| ONNXParser
    PythonDel -->|invoke delegate| TFLiteDelClassic
    PythonDel -->|invoke delegate| TFLiteDelOpaque
    PythonDel -->|invoke delegate| TFLiteDelJNI

    %% Delegate/Parser to Core
    TfLiteParser -->|parsed network| CoreRuntime
    ONNXParser -->|parsed network| CoreRuntime
    TFLiteDelClassic -->|fallback & delegate| CoreRuntime
    TFLiteDelOpaque -->|fallback & delegate| CoreRuntime
    TFLiteDelJNI -->|fallback & delegate| CoreRuntime

    %% Core to Registry to Backends
    CoreRuntime -->|register backends| BackendRegistry
    BackendRegistry -->|load plugin| RefBackend
    BackendRegistry -->|load plugin| NeonBackend
    BackendRegistry -->|load plugin| CLBackend
    BackendRegistry -->|load plugin| ACLCommon
    BackendRegistry -->|load plugin| TosaCommon
    BackendRegistry -->|load plugin| TosaRef

    %% Core to Serialization/Deserialization
    CoreRuntime ---|serialize/deserialize| Serializer
    CoreRuntime ---|serialize/deserialize| Deserializer

    %% Profiling Pipeline
    CoreRuntime -->|emit profiling| ProfClient
    ProfClient -->|use common libs| ProfCommon
    ProfCommon -->|send data| ProfServer

    %% Build Tool flows
    Docker -->|generate| CoreRuntime
    Docker -->|generate| TfLiteParser
    Docker -->|generate| ONNXParser
    Docker -->|generate| BackendRegistry
    Docker -->|generate| ProfClient

    CMake -->|configure| CoreRuntime
    CMake -->|configure| TfLiteParser
    CMake -->|configure| ONNXParser
    CMake -->|configure| BackendRegistry
    CMake -->|configure| ProfClient

    Scripts -->|support build| Docker
    Scripts -->|support build| CMake

    %% Samples & Tests link
    Samples -->|link against| CoreRuntime
    Samples -->|link against| TfLiteParser
    Samples -->|link against| ONNXParser
    Tests -->|test| CoreRuntime
    Tests -->|test| BackendRegistry

    %% Click Events
    click ClientCpp "https://github.com/arm-software/armnn/tree/main/tests/ExecuteNetwork"
    click PythonDel "https://github.com/arm-software/armnn/tree/main/delegate/python"
    click TFLiteDelClassic "https://github.com/arm-software/armnn/tree/main/delegate/classic"
    click TFLiteDelOpaque "https://github.com/arm-software/armnn/tree/main/delegate/opaque"
    click TFLiteDelJNI "https://github.com/arm-software/armnn/tree/main/delegate/armnnDelegateJNI"
    click TfLiteParser "https://github.com/arm-software/armnn/tree/main/src/armnnTfLiteParser"
    click TfLiteParser "https://github.com/arm-software/armnn/tree/main/include/armnnTfLiteParser"
    click ONNXParser "https://github.com/arm-software/armnn/tree/main/src/armnnOnnxParser"
    click ONNXParser "https://github.com/arm-software/armnn/tree/main/include/armnnOnnxParser"
    click CoreRuntime "https://github.com/arm-software/armnn/tree/main/src/armnn"
    click CoreRuntime "https://github.com/arm-software/armnn/tree/main/include/armnn"
    click BackendRegistry "https://github.com/arm-software/armnn/blob/main/include/armnn/BackendRegistry.hpp"
    click BackendRegistry "https://github.com/arm-software/armnn/blob/main/src/armnn/BackendRegistry.cpp"
    click ACLCommon "https://github.com/arm-software/armnn/blob/main/include/armnn/backends/DynamicBackend.hpp"
    click RefBackend "https://github.com/arm-software/armnn/tree/main/src/backends/reference"
    click NeonBackend "https://github.com/arm-software/armnn/tree/main/src/backends/neon"
    click CLBackend "https://github.com/arm-software/armnn/tree/main/src/backends/cl"
    click ACLCommon "https://github.com/arm-software/armnn/tree/main/src/backends/aclCommon"
    click TosaCommon "https://github.com/arm-software/armnn/tree/main/src/backends/tosaCommon"
    click TosaRef "https://github.com/arm-software/armnn/tree/main/src/backends/tosaReference"
    click Serializer "https://github.com/arm-software/armnn/tree/main/src/armnnSerializer"
    click Serializer "https://github.com/arm-software/armnn/tree/main/include/armnnSerializer"
    click Deserializer "https://github.com/arm-software/armnn/tree/main/src/armnnDeserializer"
    click Deserializer "https://github.com/arm-software/armnn/tree/main/include/armnnDeserializer"
    click ProfClient "https://github.com/arm-software/armnn/tree/main/profiling/client"
    click ProfCommon "https://github.com/arm-software/armnn/tree/main/profiling/common"
    click ProfServer "https://github.com/arm-software/armnn/tree/main/profiling/server"
    click Docker "https://github.com/arm-software/armnn/tree/main/build-tool/docker"
    click Scripts "https://github.com/arm-software/armnn/tree/main/build-tool/scripts"
    click CMake "https://github.com/arm-software/armnn/blob/main/cmake/CMakeLists.txt"
    click CMake "https://github.com/arm-software/armnn/tree/main/cmake"
    click Samples "https://github.com/arm-software/armnn/tree/main/samples/"
    click Tests "https://github.com/arm-software/armnn/tree/main/tests/"

    %% Styles
    classDef client fill:#cce5ff,stroke:#444;
    classDef core fill:#d4edda,stroke:#444;
    classDef parser fill:#ffe5b4,stroke:#444;
    classDef backend fill:#e2d6f3,stroke:#444;
    classDef profiling fill:#d1ecf1,stroke:#444;
    classDef tooling fill:#e2e3e5,stroke:#444;
```
