How pytorch/executorch works
Project Type and Purpose
• ExecuTorch is a full-stack, on-device AI framework for both inference and training.
• It provides end-to-end pipelines: model export from PyTorch to a portable format, IR transformations, code generation, hardware-specific backends, and a lightweight runtime.
• It targets mobile and embedded platforms (iOS, Android, Linux, microcontrollers) with hardware acceleration (Apple CoreML/MPS, Arm Ethos-U, Cadence, MediaTek, NXP, OpenVINO, Qualcomm, Vulkan, XNNPACK).
High-Level File Structure
• Top-Level Directories
• src/ and executorch/: Python front-end APIs, CLI tools, user examples, documentation.
• exir/: Intermediate representation (IR) library, passes, emitters.
• codegen/: Code generation from IR to C++ or platform code.
• kernel/ & kernels/: Hand-written portable, optimized, quantized kernels.
• backends/: Hardware delegates and partitioners (one subfolder per vendor).
• runtime/: Core C++ runtime, executor, memory manager, platform abstraction.
• extension/: Bindings for Android, iOS (Swift/C++/JNI) and other extension modules.
• examples/: End-to-end demos per platform and per backend.
• docs/: Sphinx-based documentation, tutorials, architecture guides.
• devtools/, tools/, profiler/: Internal tooling for debugging, visualization, CI.
• CMakeLists.txt, Package.swift, buckconfig, .github/: Build and CI config (CMake, SwiftPM, Buck2, GitHub Actions).
Architecture from the README
• ExecuTorch wraps PyTorch models into a portable PTE (Portable Executorch) format.
• The PTE is then lowered into the ExecuTorch IR (exir), transformed by passes (e.g., quantization, delegate insertion).
• codegen emits optimized C++ or platform-specific code for kernels and executors.
• kernel libs provide CPU/edge kernels; backends supply delegate implementations.
• runtime executes the generated or delegated graph on device, with a hardware-abstraction layer.
a. Main Components
• Front-end & APIs (Python/Swift/Java modules in src/, executorch/, extension/)
• Export & IR (PTE exporter → exir pipeline → transformation passes)
• Code Generation (codegen produces C++/delegate code)
• Kernel Libraries (kernels/portable, kernels/optimized, kernels/quantized)
• Backend Delegates (backends/<vendor>/..., partitioners + runtime delegates)
• Runtime Core (runtime/: executor, memory manager, platform layer)
• Extensions & Examples (platform bindings: Android JNI, iOS Swift, WASM)
• DevTools & CI (benchmarking, profiler, test suites, Docker orchestrations)
b. Relationships & Data Flow
1. User script / PyTorch model
↓ export via Python CLI
2. PTE file
↓ parsed by exir IR front-end
3. IR passes (graph transforms, quantizer, partitioner)
↓
↓
5. codegen → generate C++ code or platform binaries
↓
6. Link against kernel libs and backend delegates
↓
7. Deployed to device → runtime executes graph
↓
8. Optional profiling / debugging via DevTools
c. Architectural Patterns & Principles
• Layered architecture: Front-end → IR → Codegen → Runtime.
• Plugin/delegate pattern: Each vendor backend implements a common interface.
• IR transformation pipeline: Reusable passes for quantization, fusion, lowering.
• Separation of concerns: Python vs C++ vs platform binding layers.
• Monorepo modularity: Each component in its own directory with tests.
d. Key Technologies & Frameworks
• Languages: Python (front-end), C++ (kernels & runtime), Swift/Objective-C (iOS), Java/Kotlin (Android), FlatBuffers.
• Build: CMake, SwiftPM (Package.swift), Buck2 (.buckconfig), Bazel integrations (targets.bzl).
• CI: GitHub Actions, Docker scripts (.ci/docker), conda environments.
• Documentation: Sphinx (docs/), Markdown tutorials.