How arm-software/armnn works
Project Type and Purpose
• Arm NN is an open-source C++17 SDK—inference engine—designed to accelerate ML models (TF Lite, ONNX) on Arm CPUs, GPUs, NPUs.
• It provides:
• A core runtime for loading and executing networks.
• Parsers (TF Lite, ONNX) and a TF Lite Delegate (Python/C++).
• A plugin-style backend registry with CPU (reference, Neon), GPU (OpenCL via ACL), NPU (Ethos-N driver) backends.
• Profiling infrastructure (client, server, common).
• Serialization/deserialization, build tools, samples, tests, third-party libs.
File Structure Analysis (Top-Level)
• CMakeLists.txt, scripts, docker → build orchestration (Arm NN Build Tool).
• /, src/armnn → core runtime and network representation.
• src/parsers (armnnTfLiteParser, armnnOnnxParser), src/serializer, src/deserializer → model I/O.
• delegate/ → TF Lite Delegate (JNI, classic, opaque) and delegate/python.
• src/backends → plugin backends (aclCommon, cl, neon, reference, tosa… dynamic).
• profiling/ → profiling client, common, server components.
• samples/, tests/, docs/, third-party/ → examples, test harnesses, documentation, external libs.
README Insights
• Emphasizes: core runtime, delegate pattern for unsupported operators, parser libraries, backend plugin architecture (registry, dynamic loading), build tool for dependencies (ACL).
• Mentions Python interface for TF Lite Delegate and Debian/Android binaries.
1. Client Application
• C++ API or Python (TF Lite Delegate)
2. Delegate / Parser Layer
• TF Lite Delegate (fallback to TFLite runtime)
• TF Lite Parser, ONNX Parser
3. Arm NN Core Runtime
• Network loader, optimizer, execution frame, tensor management
4. Backend Registry & Plugin Loader
5. Compute Backends (plugin modules)
• CPU Reference, Neon
• GPU (OpenCL via ACL)
• NPU (Ethos-N driver)
• Dynamic/backendsCommon, TOSA, etc.
6. Serialization/Deserialization
• armnnSerializer, armnnDeserializer
7. Profiling Subsystem
• Profiling Client (inferred from runtime) → Profiling Connection → Profiling Server
8. Build Tool & Scripts
• Docker, bash scripts to build Arm NN + ACL + dependencies
9. Samples & Tests (execute network, delegates, backends)
• Client → Delegate/Parser (load model) → Core Runtime
• Core Runtime → Optimizer → SubgraphViewSelector → ExecutionFrame
• ExecutionFrame → Backend Registry → Selected Backend Plugin
• Backend Plugin → Hardware (CPU, GPU, NPU)
• Core Runtime ↔ Profiling Client → Profiling Connection → Profiling Server → Packet Handler
• Serialization/Deserialization components interface with Core for graph I/O.
• Build Tool → Generates libraries for core, parsers, backends, profiling.
• Samples/Tests → Link against built libraries to validate functionality.
c. Architectural Patterns and Principles
• Modular plugin architecture: backends register via registry and can be loaded dynamically.
• Delegate/Fallback pattern: TF Lite Delegate accelerates supported ops; falls back on reference runtime.
• Layered design: API → Parser/Delegate → Core Runtime → Backend → Hardware.
• Separation of concerns: profiling separated into client/common/server modules.
d. Key Technologies
• C++17, CMake, Docker, bash for build automation.
• Python bindings for TF Lite Delegate.
• Arm Compute Library (ACL) for GPU kernels.
• Ethos-N NPU driver.
• Profiling via sockets, pipe server, timeline decoder.