# zml/zml architecture

> Architecture diagram of the GitHub repository zml/zml, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zml/zml
- Repository: https://github.com/zml/zml

## Overview

ZML is a production inference stack that compiles models for multiple accelerator families. The README’s principal workflow is to provide a model and input, compile for a selected platform, load weights, and execute inference; it also documents local, Hugging Face, and S3 model sources. The supplied tree and excerpts chiefly expose MLIR dialect support, a Trainium attention kernel, profiling, and TPU metrics—not the model CLI or runtime orchestration—so those documented workflow links are not presented as inspected call paths.

Main entry points: [`mlir/dialects/cute_ir/CuteInference.cpp`](https://github.com/zml/zml/blob/master/mlir/dialects/cute_ir/CuteInference.cpp) (CuTe inference), [`bin/zml-smi/platforms/tpu/tpuinfo.cc`](https://github.com/zml/zml/blob/master/bin/zml-smi/platforms/tpu/tpuinfo.cc) (TPU metrics client), [`zml/profiling/traceme_lib.cc`](https://github.com/zml/zml/blob/master/zml/profiling/traceme_lib.cc) (TraceMe backend), [`mlir/dialects/cute_nvgpu/capi.cc`](https://github.com/zml/zml/blob/master/mlir/dialects/cute_nvgpu/capi.cc) (CuTe NVGPU API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_mlir["MLIR dialect support"]
  node_cute_infer["CuTe inference<br/>[CuteInference.cpp]"]
  node_cute_nvgpu["CuTe NVGPU API<br/>[capi.cc]"]
  node_fly_api["Fly dialect API<br/>[capi.cc]"]
  node_tcl_ops["TCL operations<br/>[tcl_ops.cc]"]
  node_air_dialect["AIR dialect<br/>[AIRDialect.cpp]"]
  node_nvvm_infer["NVVM inference<br/>[NVVMInference.cpp]"]
  node_cuda_infer["CUDA inference<br/>[CudaInference.cpp]"]
  node_cuda_tile["CUDA Tile API"]
  node_mosaic_tpu["Mosaic TPU API"]
  node_ttir_api["TTIR API"]
  node_rocdl_api["ROCDL API<br/>[capi.cc]"]
  node_fly_rocdl["Fly ROCDL API"]
end

subgraph group_accel["Accelerator support"]
  node_nki_compile["NKI compiler bridge"]
  node_tpu_metrics["TPU metrics client<br/>[tpuinfo.cc]"]
end

subgraph group_observe["Observability and metrics"]
  node_trace_api["TraceMe API<br/>[traceme.h]"]
  node_trace_impl["TraceMe backend<br/>[traceme_lib.cc]"]
  node_xprof["XProf tool<br/>[main.py]"]
end

subgraph group_kernel["Accelerator kernels"]
  node_nki_kernel["NKI attention<br/>[attention.py]"]
end

node_user(("Model user"))
node_model_flow["Compile, load, infer"]
node_metric_service["TPU metric service"]

node_user -.->|"provides model"| node_model_flow
node_model_flow -.->|"compiles with"| node_cute_infer
node_model_flow -.->|"may use"| node_nki_kernel
node_cute_infer -->|"references dialect"| node_cute_nvgpu
node_trace_impl -->|"implements API"| node_trace_api
node_tpu_metrics -->|"requests metrics"| node_metric_service

click node_cute_infer "https://github.com/zml/zml/blob/master/mlir/dialects/cute_ir/CuteInference.cpp"
click node_cute_nvgpu "https://github.com/zml/zml/blob/master/mlir/dialects/cute_nvgpu/capi.cc"
click node_fly_api "https://github.com/zml/zml/blob/master/mlir/dialects/fly/capi.cc"
click node_tcl_ops "https://github.com/zml/zml/blob/master/mlir/dialects/tcl/tcl_ops.cc"
click node_air_dialect "https://github.com/zml/zml/blob/master/mlir/dialects/air/AIRDialect.cpp"
click node_nvvm_infer "https://github.com/zml/zml/blob/master/mlir/dialects/nvvm/NVVMInference.cpp"
click node_cuda_infer "https://github.com/zml/zml/blob/master/mlir/dialects/cuda/CudaInference.cpp"
click node_cuda_tile "https://github.com/zml/zml/tree/master/mlir/dialects/cuda_tile"
click node_mosaic_tpu "https://github.com/zml/zml/tree/master/mlir/dialects/mosaic_tpu"
click node_ttir_api "https://github.com/zml/zml/tree/master/mlir/dialects/ttir"
click node_rocdl_api "https://github.com/zml/zml/blob/master/mlir/dialects/rocdl/capi.cc"
click node_fly_rocdl "https://github.com/zml/zml/tree/master/mlir/dialects/fly_rocdl"
click node_nki_kernel "https://github.com/zml/zml/blob/master/zml/kernels/flash_attention/nki/attention.py"
click node_nki_compile "https://github.com/zml/zml/blob/master/platforms/neuron/nki_kernel_compiler.py"
click node_trace_api "https://github.com/zml/zml/blob/master/zml/profiling/traceme.h"
click node_trace_impl "https://github.com/zml/zml/blob/master/zml/profiling/traceme_lib.cc"
click node_xprof "https://github.com/zml/zml/blob/master/tools/xprof/main.py"
click node_tpu_metrics "https://github.com/zml/zml/blob/master/bin/zml-smi/platforms/tpu/tpuinfo.cc"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cute_infer,node_cute_nvgpu,node_fly_api,node_tcl_ops,node_air_dialect,node_nvvm_infer,node_cuda_infer,node_cuda_tile,node_mosaic_tpu,node_ttir_api,node_rocdl_api,node_fly_rocdl,node_user toneBlue
class node_nki_compile,node_tpu_metrics toneAmber
class node_trace_api,node_trace_impl,node_xprof toneMint
class node_nki_kernel toneRose
class node_model_flow,node_metric_service toneIndigo
```

## Components

### MLIR dialect support

- [CuTe inference](https://github.com/zml/zml/blob/master/mlir/dialects/cute_ir/CuteInference.cpp): `mlir/dialects/cute_ir/CuteInference.cpp`
- [CuTe NVGPU API](https://github.com/zml/zml/blob/master/mlir/dialects/cute_nvgpu/capi.cc): `mlir/dialects/cute_nvgpu/capi.cc`
- [Fly dialect API](https://github.com/zml/zml/blob/master/mlir/dialects/fly/capi.cc): `mlir/dialects/fly/capi.cc`
- [TCL operations](https://github.com/zml/zml/blob/master/mlir/dialects/tcl/tcl_ops.cc): `mlir/dialects/tcl/tcl_ops.cc`
- [AIR dialect](https://github.com/zml/zml/blob/master/mlir/dialects/air/AIRDialect.cpp): `mlir/dialects/air/AIRDialect.cpp`
- [NVVM inference](https://github.com/zml/zml/blob/master/mlir/dialects/nvvm/NVVMInference.cpp): `mlir/dialects/nvvm/NVVMInference.cpp`
- [CUDA inference](https://github.com/zml/zml/blob/master/mlir/dialects/cuda/CudaInference.cpp): `mlir/dialects/cuda/CudaInference.cpp`
- [CUDA Tile API](https://github.com/zml/zml/tree/master/mlir/dialects/cuda_tile): `mlir/dialects/cuda_tile`
- [Mosaic TPU API](https://github.com/zml/zml/tree/master/mlir/dialects/mosaic_tpu): `mlir/dialects/mosaic_tpu`
- [TTIR API](https://github.com/zml/zml/tree/master/mlir/dialects/ttir): `mlir/dialects/ttir`
- [ROCDL API](https://github.com/zml/zml/blob/master/mlir/dialects/rocdl/capi.cc): `mlir/dialects/rocdl/capi.cc`
- [Fly ROCDL API](https://github.com/zml/zml/tree/master/mlir/dialects/fly_rocdl): `mlir/dialects/fly_rocdl`

### Accelerator support

- [NKI compiler bridge](https://github.com/zml/zml/blob/master/platforms/neuron/nki_kernel_compiler.py): `platforms/neuron/nki_kernel_compiler.py`
- [TPU metrics client](https://github.com/zml/zml/blob/master/bin/zml-smi/platforms/tpu/tpuinfo.cc): `bin/zml-smi/platforms/tpu/tpuinfo.cc`

### Observability and metrics

- [TraceMe API](https://github.com/zml/zml/blob/master/zml/profiling/traceme.h): `zml/profiling/traceme.h`
- [TraceMe backend](https://github.com/zml/zml/blob/master/zml/profiling/traceme_lib.cc): `zml/profiling/traceme_lib.cc`
- [XProf tool](https://github.com/zml/zml/blob/master/tools/xprof/main.py): `tools/xprof/main.py`

### Accelerator kernels

- [NKI attention](https://github.com/zml/zml/blob/master/zml/kernels/flash_attention/nki/attention.py): `zml/kernels/flash_attention/nki/attention.py`

### Other components

- **Model user**
- **Compile, load, infer**
- **TPU metric service**

## Connections

- Model user → Compile, load, infer: provides model (evidence: `README.md`)
- Compile, load, infer → CuTe inference: compiles with (evidence: `README.md`)
- Compile, load, infer → NKI attention: may use (evidence: `README.md`)
- CuTe inference → CuTe NVGPU API: references dialect (evidence: `mlir/dialects/cute_ir/CuteInference.cpp`)
- TraceMe backend → TraceMe API: implements API (evidence: `zml/profiling/traceme_lib.cc`)
- TPU metrics client → TPU metric service: requests metrics (evidence: `bin/zml-smi/platforms/tpu/tpuinfo.cc`)
