How maderix/ane works
Project type + purpose (what you’re diagramming)
This is a native macOS (Objective‑C/C) research project that implements transformer training (forward + backward) on Apple Neural Engine (ANE) using reverse‑engineered private APIs. It is *not* a typical “service architecture”; the “system” is a single-process training runtime that orchestrates:
• MIL graph generation (program text)
• In-memory ANE compilation (private _ANEClient/_ANECompiler pathways)
• ANE kernel execution (6 compiled kernels per training step)
• IOSurface tensor I/O (shared memory)
• CPU-side math (RMSNorm backward, residuals, loss, dW via Accelerate/CBLAS, Adam)
• Concurrency/overlap (GCD to overlap CPU GEMMs with ANE eval)
• Tests/benchmarks (microbench and kernel-specific tests)
Top-level (exploration/bench layer)
• inmem_basic.m, inmem_bench.m, inmem_peak.m — in-memory compile + latency + peak TFLOPS tooling
• sram_bench.m, sram_probe.m — ANE SRAM characterization
These are adjacent utilities to the core training system; show them as “Tooling / Benchmarks” feeding knowledge into the runtime, not as runtime dependencies.
Core training runtime (inside training/)
• ane_runtime.h — private API wrapper: compile/eval, IOSurface management, runtime calling via objc_msgSend
• ane_mil_gen.h — MIL program generation helpers (string/IR builders)
• model.h — weights init, blob builders (BLOBFILE constants)
• forward.h — forward MIL generators
• backward.h — backward MIL generators
• train_large.m — main optimized training loop (the “product”)
• test_*.m — kernel unit tests (validation layer)
• dashboard.py, tokenize.py — peripheral (metrics/UI + WIP data pipeline)
• Makefile, training/README.md — build/docs (not runtime)
Training step = 6 ANE kernels per step
These are “compiled programs” executed on ANE:
CPU responsibilities
• RMSNorm backward (vDSP)
• residual connections
• loss
• dW gradients via cblas_sgemm (Accelerate)
• Adam optimizer updates
• gradient accumulation, checkpoint/resume
• orchestration and scheduling (GCD overlap with ANE)
Data representation + I/O boundary
• IOSurface tensors in [1, C, 1, S] channel-first fp16 layout
• “Forward taps” exposing intermediates via concat outputs (so backward can run without recompute)
Compilation model
• MIL generated at runtime
• _ANEInMemoryModelDescriptor compiles MIL+weight blobs directly
• Weights are embedded; weight changes imply recompilation (“recompiled each batch when weights change”)
• exec() restart to bypass compiler resource leak/compile limit
These constraints should appear either in callouts or in an “Important constraints/notes” sidebar.
Orchestration / Control (CPU process)
• Training loop (train_large.m)
• Scheduler (GCD queues)
• Checkpoint/Resume
• Metrics/logging (optional)
Compilation & Runtime Interface
• MIL generators (ane_mil_gen.h, forward.h, backward.h)
• Weight/blob builder (model.h)
• ANE runtime wrapper (ane_runtime.h)
Execution Substrates
• ANE hardware executing 6 kernels
• IOSurface shared memory as the tensor transport
• Accelerate framework (CBLAS/vDSP) for CPU math
This immediately communicates “single process, multi-subsystem”.
Make the 6-kernel step explicit as a pipeline (core
kFwdAttn → kFwdFFN → kFFNBwd → kSdpaBwd1 → kSdpaBwd2 → kQKVb
• inputs (e.g., X, weights, mask, tapped tensors)
• outputs (e.g., hidden state, probs, dQ/dK/dV, dx)
• whether it consumes embedded weights (most do)
• IOSurface buffers (data plane)
• CPU consumers (loss, RMSNorm backward, dW GEMMs)
Compile-time vs run-time flows (very important here)
Two distinct flows:
Compile flow (control plane)
train_large.m
→ calls MIL builders (forward.h/backward.h + ane_mil_gen.h)
→ emits MIL text + constant blobs (model.h)
→ via runtime wrapper (ane_runtime.h)
→ private ANE compiler APIs
→ produces executable ANE “programs” (the 6 kernels)
Execute flow (data plane)
CPU prepares IOSurfaces (inputs, activations, grads)
→ ane_runtime.h eval()
→ ANE reads/writes IOSurface buffers
→ CPU reads taps/outputs for dW, optimizer, etc.
CPU/ANE overlap (a performance-critical relationship)
• It runs concurrently with ANE eval using GCD async
• “Deferred cblas wait” (the synchronization point pushed into next step)
High-level runtime architecture (single-process)
Shows CPU training loop, MIL generation, ANE compile/eval, IOSurface, Accelerate, and private API boundary.
Per-step kernel pipeline + CPU responsibilities
Shows the 6 kernels in order, the key tensors/taps between them, and the CPU-side tasks (loss, RMSNorm bwd, dW GEMMs, Adam) with overlap.
A small third:
Tooling/benchmarks map
Shows top-level inmem_* and sram_* programs as “research harnesses” around the core runtime.
Don’t overthink it: split into as many components as possible
• separate “MIL generation” from “weight blob building”
• separate “compile” from “eval”
• separate “IOSurface allocation” from “IOSurface data”
• separate each of the 6 kernels
• separate “CPU dW GEMM” from “CPU optimizer”
• separate “tests” and “benchmarks”