# maderix/ane architecture

> Architecture diagram of the GitHub repository maderix/ane, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/maderix/ane
- Repository: https://github.com/maderix/ane

## Overview

##### Project type + purpose (what you’re diagramming)
This is **a native macOS (Objective‑C/C) research project** that implements **transformer training (forward + backward) on Apple Neural Engine (ANE)** using **reverse‑engineered private APIs**. It is *not* a typical “service architecture”; the “system” is a **single-process training runtime** that orchestrates:

- **MIL graph generation** (program text)
- **In-memory ANE compilation** (private `_ANEClient/_ANECompiler` pathways)
- **ANE kernel execution** (6 compiled kernels per training step)
- **IOSurface tensor I/O** (shared memory)
- **CPU-side math** (RMSNorm backward, residuals, loss, dW via Accelerate/CBLAS, Adam)
- **Concurrency/overlap** (GCD to overlap CPU GEMMs with ANE eval)
- **Tests/benchmarks** (microbench and kernel-specific tests)

##### Top-level (exploration/bench layer)
- `inmem_basic.m`, `inmem_bench.m`, `inmem_peak.m` — in-memory compile + latency + peak TFLOPS tooling
- `sram_bench.m`, `sram_probe.m` — ANE SRAM characterization

These are **adjacent utilities** to the core training system; show them as “Tooling / Benchmarks” feeding knowledge into the runtime, not as runtime dependencies.

##### Core training runtime (inside `training/`)
- `ane_runtime.h` — **private API wrapper**: compile/eval, IOSurface management, runtime calling via `objc_msgSend`
- `ane_mil_gen.h` — **MIL program generation** helpers (string/IR builders)
- `model.h` — weights init, blob builders (BLOBFILE constants)
- `forward.h` — **forward MIL generators**
- `backward.h` — **backward MIL generators**
- `train_large.m` — main optimized training loop (the “product”)
- `test_*.m` — kernel unit tests (validation layer)
- `dashboard.py`, `tokenize.py` — peripheral (metrics/UI + WIP data pipeline)
- `Makefile`, `training/README.md` — build/docs (not runtime)

##### Training step = 6 ANE kernels per step
These are “compiled programs” executed on ANE:

##### CPU responsibilities
- RMSNorm backward (vDSP)
- residual connections
- loss
- **dW gradients** via `cblas_sgemm` (Accelerate)
- Adam optimizer updates
- gradient accumulation, checkpoint/resume
- orchestration and scheduling (GCD overlap with ANE)

##### Data representation + I/O boundary
- IOSurface tensors in `[1, C, 1, S]` channel-first fp16 layout
- “Forward taps” exposing intermediates via concat outputs (so backward can run without recompute)

##### Compilation model
- MIL generated at runtime
- `_ANEInMemoryModelDescriptor` compiles MIL+weight blobs directly
- Weights are embedded; weight changes imply recompilation (“recompiled each batch when weights change”)
- `exec() restart` to bypass compiler resource leak/compile limit

These constraints should appear either in callouts or in an “Important constraints/notes” sidebar.

##### Orchestration / Control (CPU process)
- Training loop (`train_large.m`)
- Scheduler (GCD queues)
- Checkpoint/Resume
- Metrics/logging (optional)

##### Compilation & Runtime Interface
- MIL generators (`ane_mil_gen.h`, `forward.h`, `backward.h`)
- Weight/blob builder (`model.h`)
- ANE runtime wrapper (`ane_runtime.h`)

##### Execution Substrates
- **ANE hardware** executing 6 kernels
- **IOSurface shared memory** as the tensor transport
- **Accelerate framework** (CBLAS/vDSP) for CPU math

This immediately communicates “single process, multi-subsystem”.

##### Make the 6-kernel step explicit as a pipeline (core

`kFwdAttn → kFwdFFN → kFFNBwd → kSdpaBwd1 → kSdpaBwd2 → kQKVb`

- **inputs** (e.g., X, weights, mask, tapped tensors)
- **outputs** (e.g., hidden state, probs, dQ/dK/dV, dx)
- whether it **consumes embedded weights** (most do)

- IOSurface buffers (data plane)
- CPU consumers (loss, RMSNorm backward, dW GEMMs)

##### Compile-time vs run-time flows (very important here)

Two distinct flows:

##### Compile flow (control plane)
`train_large.m`
→ calls MIL builders (`forward.h/backward.h` + `ane_mil_gen.h`)
→ emits MIL text + constant blobs (`model.h`)
→ via runtime wrapper (`ane_runtime.h`)
→ private ANE compiler APIs
→ produces executable ANE “programs” (the 6 kernels)

##### Execute flow (data plane)
CPU prepares IOSurfaces (inputs, activations, grads)
→ `ane_runtime.h` eval()
→ ANE reads/writes IOSurface buffers
→ CPU reads taps/outputs for dW, optimizer, etc.

##### CPU/ANE overlap (a performance-critical relationship)
- It runs **concurrently** with ANE eval using **GCD async**
- “Deferred cblas wait” (the synchronization point pushed into next step)

##### High-level runtime architecture (single-process)
Shows CPU training loop, MIL generation, ANE compile/eval, IOSurface, Accelerate, and private API boundary.

##### Per-step kernel pipeline + CPU responsibilities
Shows the 6 kernels in order, the key tensors/taps between them, and the CPU-side tasks (loss, RMSNorm bwd, dW GEMMs, Adam) with overlap.

A small third:
##### Tooling/benchmarks map
Shows top-level `inmem_*` and `sram_*` programs as “research harnesses” around the core runtime.

##### Don’t overthink it: split into as many components as possible
- separate “MIL generation” from “weight blob building”
- separate “compile” from “eval”
- separate “IOSurface allocation” from “IOSurface data”
- separate each of the 6 kernels
- separate “CPU dW GEMM” from “CPU optimizer”
- separate “tests” and “benchmarks”

## Diagram (Mermaid)

```mermaid
flowchart TD

%% =========================================================
%% Diagram 1: High-level runtime architecture (single-process)
%% =========================================================
subgraph D1["Diagram 1 — High-level runtime architecture (single-process)"]
direction TB

subgraph CPU1["Orchestration / Control (CPU process)"]
direction TB
TL["Training Loop (train_large.m)"]:::cpu
SCH["Scheduler / Overlap (GCD async)"]:::cpu
CKPT["Checkpoint/Resume + exec() restart\n(bypass compiler leak/compile limit)"]:::cpu
MET["Metrics/Logging (dashboard.py)"]:::tool
DATA["Tokenization / WIP data pipeline (tokenize.py)"]:::tool

BASE["Baseline loops (train.m, tiny_train.m, tiny_train_old.m)"]:::cpu
end

subgraph CRI1["Compilation & Runtime Interface (control plane + data plane boundary)"]
direction TB
MILGEN["MIL Generation\n(ane_mil_gen.h + forward.h + backward.h)"]:::compile
WB["Weights/Blobs Builder\n(model.h, BLOBFILE constants)"]:::compile
STMIL["Stories: MIL variants/glue\n(stories_mil.h)"]:::compile
STIO["Stories: I/O + IOSurface plumbing\n(stories_io.h)"]:::runtime
STCPU["Stories: CPU ops helpers\n(stories_cpu_ops.h)"]:::cpu
STCFG["Stories: Run config/constants\n(stories_config.h)"]:::cpu

RT["ANE Runtime Wrapper\n(ane_runtime.h)\ncompile/eval + IOSurface mgmt + objc_msgSend"]:::runtime

subgraph PRIVB["Private/Undocumented ANE Boundary (dashed)"]
direction TB
PRIV["Private ANE APIs\n(_ANEClient/_ANECompiler/_ANEInMemoryModelDescriptor)"]:::private
end
end

subgraph SUB1["Execution Substrates (data plane)"]
direction TB
IOS["IOSurface tensors\nshared memory buffers\nlayout: [1,C,1,S] fp16 (CHW-ish)"]:::ios
ANE["Apple Neural Engine (ANE)\nexecutes compiled kernels"]:::ane
ACC["Accelerate Framework\nCBLAS (sgemm) + vDSP (RMSNorm bwd)"]:::accel
end

%% --- Control plane (compile) ---
TL -.->|"generate MIL"| MILGEN
TL -.->|"build blobs/constants"| WB
MILGEN -.->|"emit MIL text"| RT
WB -.->|"embed weights into program blobs"| RT
STMIL -.->|"select variants"| MILGEN
STCFG -.->|"configure runs"| TL
RT -.->|"compile(in-memory model)"| PRIV
PRIV -.->|"produce executable programs\n(6 kernels/step)"| RT

%% --- Data plane (execute) ---
TL -->|"allocate/prepare IOSurfaces"| IOS
STIO -->|"tensor views + surface plumbing"| IOS
TL -->|"eval(programs) + bind IOSurfaces"| RT
RT -->|"eval() dispatch"| PRIV
PRIV -->|"run on device"| ANE
ANE -->|"read/write IOSurface"| IOS

%% --- CPU math responsibilities ---
IOS -->|"read taps/activations/grads"| TL
TL -->|"RMSNorm backward (vDSP)"| ACC
TL -->|"dW gradients (cblas_sgemm)"| ACC
TL -->|"Adam update + grad accum"| ACC
SCH -.->|"overlap CPU GEMMs with ANE eval"| TL
CKPT -.->|"checkpoint / resume"| TL

%% --- Peripheral / optional connections ---
DATA -.->|"tokens/batches (WIP)"| TL
MET -.->|"plots/telemetry"| TL
BASE -.->|"reference/earlier loops"| TL
end

%% ==============================================
%% Diagram 2: Per-step kernel pipeline + CPU work
%% ==============================================
subgraph D2["Diagram 2 — Per-step training step: 6 ANE kernels + CPU responsibilities"]
direction TB

subgraph STEP["One Training Step (single process)"]
direction TB

subgraph HOST["Host (CPU) — orchestration + math"]
direction TB
STEPCTRL["Step Orchestrator\n(train_large.m)"]:::cpu
CPU_RES["CPU ops:\n- residual connections\n- loss\n- RMSNorm backward (vDSP)\n- dW GEMMs (cblas_sgemm)\n- Adam update\n- grad accumulation"]:::cpu
SYNC["Deferred wait / sync point\n(pushed into next step)"]:::cpu
end

subgraph IO2["Tensor Transport"]
direction TB
IOS2["IOSurface buffers\ninputs/activations/grads\n[1,C,1,S] fp16"]:::ios
TAPS["Forward taps (concat outputs)\nexpose intermediates for backward\n(no recompute)"]:::ios
end

subgraph DEV["Device (ANE) — compiled programs"]
direction TB
K1["kFwdAttn\nRMSNorm + QKV proj + SDPA + out proj"]:::kernel
K2["kFwdFFN\nRMSNorm + SwiGLU FFN"]:::kernel
K3["kFFNBwd\nFFN backward → dx parts"]:::kernel
K4["kSdpaBwd1\nWoᵀ + SDPA bwd (part 1)"]:::kernel
K5["kSdpaBwd2\nSDPA bwd (part 2)"]:::kernel
K6["kQKVb\nQKV backward → dx"]:::kernel
end

subgraph COMP2["Compile/Eval Interface"]
direction TB
RT2["ane_runtime.h\n(eval + IOSurface binding)"]:::runtime
PRIV2["Private ANE APIs\n(_ANEClient/_ANECompiler/_ANEInMemoryModelDescriptor)\nweights baked-in ⇒ weight update => recompile"]:::private
end

%% --- Execute flow (data plane, solid) ---
STEPCTRL -->|"prepare inputs + bind surfaces"| IOS2
IOS2 -->|"X/activations"| K1
K1 -->|"hidden state + taps"| K2
K2 -->|"hidden state + taps"| K3
K3 -->|"grads/taps"| K4
K4 -->|"grads"| K5
K5 -->|"grads"| K6
K6 -->|"dx/grads"| IOS2

K1 -->|"emit forward taps"| TAPS
K2 -->|"emit forward taps"| TAPS

%% --- Runtime dispatch to ANE ---
STEPCTRL -->|"eval(program sequence)"| RT2
RT2 -->|"objc_msgSend into private APIs"| PRIV2
PRIV2 -->|"dispatch kernels"| DEV

%% --- CPU consumption + overlap ---
TAPS -->|"consume forward taps"| CPU_RES
IOS2 -->|"read activations/grads"| CPU_RES
CPU_RES -->|"write updated weights/grads"| IOS2

STEPCTRL -.->|"GCD async overlap"| CPU_RES
DEV -.->|"runs concurrently with"| CPU_RES
CPU_RES -.->|"defer cblas wait"| SYNC
SYNC -.->|"next step barrier"| STEPCTRL
end
end

%% ===================================
%% Diagram 3: Tooling / benchmarks map
%% ===================================
subgraph D3["Diagram 3 — Research tooling / benchmarks (adjacent to runtime)"]
direction TB

subgraph TOOLS["Tooling / Benchmarks (not runtime deps)"]
direction TB
APIEXP["ANE API discovery\n(api_exploration.m)"]:::tool
INBASIC["In-memory compile + latency tool\n(inmem_basic.m)"]:::tool
INBENCH["In-memory compile benchmark harness\n(inmem_bench.m)"]:::tool
INPEAK["Peak TFLOPS / throughput benchmark\n(inmem_peak.m)"]:::tool
SRBENCH["ANE SRAM benchmark\n(sram_bench.m)"]:::tool
SRPROBE["ANE SRAM probing/characterization\n(sram_probe.m)"]:::tool
end

subgraph CORE["Core training runtime (training/)"]
direction TB
TRAINBOX["Training runtime\n(train_large.m + training/ headers)"]:::cpu
DOCS["Docs/Build\n(training/README.md, training/Makefile)"]:::doc
TESTS["Kernel/unit tests (training/test_*.m)"]:::test
end

TOOLS -.->|"reverse-engineer APIs\n+ characterize perf"| TRAINBOX
TOOLS -.->|"inform constraints/tuning"| TESTS
DOCS -.->|"guides usage + architecture"| TRAINBOX
TESTS -.->|"validate kernels + behavior"| TRAINBOX
end

%% ======================
%% Click events (mapping)
%% ======================
click APIEXP "https://github.com/maderix/ane/blob/main/api_exploration.m"
click INBASIC "https://github.com/maderix/ane/blob/main/inmem_basic.m"
click INBENCH "https://github.com/maderix/ane/blob/main/inmem_bench.m"
click INPEAK "https://github.com/maderix/ane/blob/main/inmem_peak.m"
click SRBENCH "https://github.com/maderix/ane/blob/main/sram_bench.m"
click SRPROBE "https://github.com/maderix/ane/blob/main/sram_probe.m"

click TL "https://github.com/maderix/ane/blob/main/training/train_large.m"
click BASE "https://github.com/maderix/ane/blob/main/training/train.m"
click BASE "https://github.com/maderix/ane/blob/main/training/tiny_train.m"
click BASE "https://github.com/maderix/ane/blob/main/training/tiny_train_old.m"

click RT "https://github.com/maderix/ane/blob/main/training/ane_runtime.h"
click MILGEN "https://github.com/maderix/ane/blob/main/training/ane_mil_gen.h"
click MILGEN "https://github.com/maderix/ane/blob/main/training/forward.h"
click MILGEN "https://github.com/maderix/ane/blob/main/training/backward.h"
click WB "https://github.com/maderix/ane/blob/main/training/model.h"

click STMIL "https://github.com/maderix/ane/blob/main/training/stories_mil.h"
click STIO "https://github.com/maderix/ane/blob/main/training/stories_io.h"
click STCPU "https://github.com/maderix/ane/blob/main/training/stories_cpu_ops.h"
click STCFG "https://github.com/maderix/ane/blob/main/training/stories_config.h"

click MET "https://github.com/maderix/ane/blob/main/training/dashboard.py"
click DATA "https://github.com/maderix/ane/blob/main/training/tokenize.py"

click TESTS "https://github.com/maderix/ane/blob/main/training/test_ane_advanced.m"
click TESTS "https://github.com/maderix/ane/blob/main/training/test_ane_causal_attn.m"
click TESTS "https://github.com/maderix/ane/blob/main/training/test_ane_sdpa5.m"
click TESTS "https://github.com/maderix/ane/blob/main/training/test_conv_attn3.m"
click TESTS "https://github.com/maderix/ane/blob/main/training/test_full_fused.m"
click TESTS "https://github.com/maderix/ane/blob/main/training/test_fused_bwd.m"
click TESTS "https://github.com/maderix/ane/blob/main/training/test_fused_qkv.m"
click TESTS "https://github.com/maderix/ane/blob/main/training/test_perf_stats.m"
click TESTS "https://github.com/maderix/ane/blob/main/training/test_qos_sweep.m"
click TESTS "https://github.com/maderix/ane/blob/main/training/test_weight_reload.m"

click DOCS "https://github.com/maderix/ane/tree/main/training/Makefile"
click DOCS "https://github.com/maderix/ane/blob/main/training/README.md"

%% =========
%% Styling
%% =========
classDef cpu fill:#E3F2FD,stroke:#1565C0,stroke-width:1.5px,color:#0D47A1
classDef compile fill:#E8F5E9,stroke:#2E7D32,stroke-width:1.5px,color:#1B5E20
classDef runtime fill:#FFF3E0,stroke:#EF6C00,stroke-width:1.5px,color:#E65100
classDef private fill:#FCE4EC,stroke:#AD1457,stroke-width:2px,stroke-dasharray: 6 4,color:#880E4F
classDef ane fill:#E0F7FA,stroke:#00838F,stroke-width:2px,color:#004D40
classDef ios fill:#F3E5F5,stroke:#6A1B9A,stroke-width:1.5px,color:#4A148C
classDef accel fill:#FFFDE7,stroke:#F9A825,stroke-width:1.5px,color:#8D6E00
classDef kernel fill:#E0F2F1,stroke:#00796B,stroke-width:1.5px,color:#004D40
classDef tool fill:#ECEFF1,stroke:#455A64,stroke-width:1.5px,color:#263238
classDef test fill:#FFEBEE,stroke:#C62828,stroke-width:1.5px,color:#B71C1C
classDef doc fill:#F1F8E9,stroke:#558B2F,stroke-width:1.5px,color:#33691E
```
