# mivey/hls_llama architecture

> Architecture diagram of the GitHub repository mivey/hls_llama, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mivey/hls_llama
- Repository: https://github.com/mivey/hls_llama

## Overview

HLS_llama is a Versal VE2302 accelerator and A72 host for int8 LLaMA-2 inference. The user supplies a prompt and checkpoint; the host tokenizes the prompt, prepares model weights and drives prefill/decode steps through either of two engines. A single programmable-logic kernel schedules transformer stages, updates the KV cache and samples tokens, which the host decodes for output. The README documents the reference C inference path and major kernel optimizations; implementation links for some host workflow details are inferred from the documented module roles, since their source was not sampled.

Main entry points: [`accelerated_llama/fastforward.cpp`](https://github.com/mivey/hls_llama/blob/main/accelerated_llama/fastforward.cpp) (Generation Driver), [`transformer_cu/transformer_kernel.cpp`](https://github.com/mivey/hls_llama/blob/main/transformer_cu/transformer_kernel.cpp) (State Selector), [`accelerated_llama/generate_loop.h`](https://github.com/mivey/hls_llama/blob/main/accelerated_llama/generate_loop.h) (Prefill Decode Loop), [`accelerated_llama/compute_units.h`](https://github.com/mivey/hls_llama/blob/main/accelerated_llama/compute_units.h) (IP / XRT Engines)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_host["Host Inference"]
  node_host_app["Generation Driver<br/>[fastforward.cpp]"]
  node_tokenizer["BPE Tokenizer<br/>[fastforward.cpp]"]
  node_weights["Checkpoint Repacker<br/>[compute_units.h]"]
  node_gen_loop["Prefill Decode Loop<br/>[generate_loop.h]"]
  node_engines["IP / XRT Engines<br/>[compute_units.h]"]
end

subgraph group_kernel["Transformer Kernel"]
  node_kernel_top["Transformer Kernel"]
  node_fsm["49-Step FSM"]
  node_prefetch["Weight Prefetch"]
  node_selector["State Selector"]
end

subgraph group_compute["Compute Stages"]
  node_rms["RMSNorm Residual<br/>[rmsnorm.cpp]"]
  node_mha["RoPE Attention<br/>[mha.cpp]"]
  node_quantizer["Int8 Quantizer<br/>[quantizer.cpp]"]
  node_gemv["Split GeMV<br/>[matmult.cpp]"]
  node_swiglu["SwiGLU<br/>[swiglu_kernel.cpp]"]
  node_sampler["Top-64 Sampler<br/>[combiner.h]"]
end

subgraph group_state["Model State"]
  node_token_state[("Token Buffer")]
  node_kv_cache[("KV Cache<br/>[mha.cpp]")]
  node_weight_store[("DDR Weights")]
end

subgraph group_reference["Reference Inference"]
  node_software_ref["Int8 C Reference<br/>[runq.c]"]
end

node_user(("Prompt User"))
node_device["Versal VE2302"]

node_user -->|"provides prompt"| node_host_app
node_host_app -->|"tokenizes"| node_tokenizer
node_host_app -->|"loads checkpoint"| node_weights
node_host_app -->|"starts generation"| node_gen_loop
node_gen_loop -->|"drives steps"| node_engines
node_engines -->|"starts kernel"| node_device
node_weights -->|"loads buffers"| node_device
node_device -->|"executes"| node_kernel_top
node_kernel_top -->|"builds schedule"| node_fsm
node_kernel_top -->|"streams weights"| node_prefetch
node_fsm -->|"dispatches states"| node_selector
node_prefetch -->|"feeds weights"| node_gemv
node_selector -->|"invokes"| node_rms
node_selector -->|"invokes"| node_mha
node_selector -->|"invokes"| node_swiglu
node_selector -->|"invokes"| node_quantizer
node_quantizer -->|"feeds activations"| node_gemv
node_gemv -->|"streams logits"| node_sampler
node_gemv -->|"writes activations"| node_token_state
node_sampler -->|"writes sampled token"| node_token_state
node_mha -->|"reads and updates"| node_kv_cache
node_weight_store -->|"reads weights"| node_prefetch
node_token_state -->|"reads token"| node_kernel_top
node_kernel_top -->|"returns token"| node_engines
node_engines -->|"returns token"| node_gen_loop
node_gen_loop -->|"emits decoded output"| node_host_app
node_tokenizer -->|"decodes token"| node_host_app
node_software_ref -.->|"reference behavior"| node_host_app

click node_host_app "https://github.com/mivey/hls_llama/blob/main/accelerated_llama/fastforward.cpp"
click node_tokenizer "https://github.com/mivey/hls_llama/blob/main/accelerated_llama/fastforward.cpp"
click node_weights "https://github.com/mivey/hls_llama/blob/main/accelerated_llama/compute_units.h"
click node_gen_loop "https://github.com/mivey/hls_llama/blob/main/accelerated_llama/generate_loop.h"
click node_engines "https://github.com/mivey/hls_llama/blob/main/accelerated_llama/compute_units.h"
click node_kernel_top "https://github.com/mivey/hls_llama/blob/main/transformer_cu/transformer_kernel.cpp"
click node_fsm "https://github.com/mivey/hls_llama/blob/main/transformer_cu/transformer_kernel.cpp"
click node_prefetch "https://github.com/mivey/hls_llama/blob/main/transformer_cu/transformer_kernel.cpp"
click node_selector "https://github.com/mivey/hls_llama/blob/main/transformer_cu/transformer_kernel.cpp"
click node_rms "https://github.com/mivey/hls_llama/blob/main/transformer_cu/rmsnorm.cpp"
click node_mha "https://github.com/mivey/hls_llama/blob/main/transformer_cu/mha.cpp"
click node_quantizer "https://github.com/mivey/hls_llama/blob/main/transformer_cu/quantizer.cpp"
click node_gemv "https://github.com/mivey/hls_llama/blob/main/transformer_cu/matmult.cpp"
click node_swiglu "https://github.com/mivey/hls_llama/blob/main/transformer_cu/swiglu_kernel.cpp"
click node_sampler "https://github.com/mivey/hls_llama/blob/main/transformer_cu/combiner.h"
click node_token_state "https://github.com/mivey/hls_llama/blob/main/transformer_cu/transformer_kernel.cpp"
click node_kv_cache "https://github.com/mivey/hls_llama/blob/main/transformer_cu/mha.cpp"
click node_weight_store "https://github.com/mivey/hls_llama/blob/main/transformer_cu/transformer_kernel.cpp"
click node_software_ref "https://github.com/mivey/hls_llama/blob/main/llama2_software/runq.c"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_host_app,node_tokenizer,node_weights,node_gen_loop,node_engines,node_user toneBlue
class node_kernel_top,node_fsm,node_prefetch,node_selector toneAmber
class node_rms,node_mha,node_quantizer,node_gemv,node_swiglu,node_sampler toneMint
class node_token_state,node_kv_cache,node_weight_store toneRose
class node_software_ref,node_device toneIndigo
```

## Components

### Host Inference

- [Generation Driver](https://github.com/mivey/hls_llama/blob/main/accelerated_llama/fastforward.cpp): `accelerated_llama/fastforward.cpp`
- [BPE Tokenizer](https://github.com/mivey/hls_llama/blob/main/accelerated_llama/fastforward.cpp): `accelerated_llama/fastforward.cpp`
- [Checkpoint Repacker](https://github.com/mivey/hls_llama/blob/main/accelerated_llama/compute_units.h): `accelerated_llama/compute_units.h`
- [Prefill Decode Loop](https://github.com/mivey/hls_llama/blob/main/accelerated_llama/generate_loop.h): `accelerated_llama/generate_loop.h`
- [IP / XRT Engines](https://github.com/mivey/hls_llama/blob/main/accelerated_llama/compute_units.h): `accelerated_llama/compute_units.h`

### Transformer Kernel

- [Transformer Kernel](https://github.com/mivey/hls_llama/blob/main/transformer_cu/transformer_kernel.cpp): `transformer_cu/transformer_kernel.cpp`
- [49-Step FSM](https://github.com/mivey/hls_llama/blob/main/transformer_cu/transformer_kernel.cpp): `transformer_cu/transformer_kernel.cpp`
- [Weight Prefetch](https://github.com/mivey/hls_llama/blob/main/transformer_cu/transformer_kernel.cpp): `transformer_cu/transformer_kernel.cpp`
- [State Selector](https://github.com/mivey/hls_llama/blob/main/transformer_cu/transformer_kernel.cpp): `transformer_cu/transformer_kernel.cpp`

### Compute Stages

- [RMSNorm Residual](https://github.com/mivey/hls_llama/blob/main/transformer_cu/rmsnorm.cpp): `transformer_cu/rmsnorm.cpp`
- [RoPE Attention](https://github.com/mivey/hls_llama/blob/main/transformer_cu/mha.cpp): `transformer_cu/mha.cpp`
- [Int8 Quantizer](https://github.com/mivey/hls_llama/blob/main/transformer_cu/quantizer.cpp): `transformer_cu/quantizer.cpp`
- [Split GeMV](https://github.com/mivey/hls_llama/blob/main/transformer_cu/matmult.cpp): `transformer_cu/matmult.cpp`
- [SwiGLU](https://github.com/mivey/hls_llama/blob/main/transformer_cu/swiglu_kernel.cpp): `transformer_cu/swiglu_kernel.cpp`
- [Top-64 Sampler](https://github.com/mivey/hls_llama/blob/main/transformer_cu/combiner.h): `transformer_cu/combiner.h`

### Model State

- [Token Buffer](https://github.com/mivey/hls_llama/blob/main/transformer_cu/transformer_kernel.cpp): `transformer_cu/transformer_kernel.cpp`
- [KV Cache](https://github.com/mivey/hls_llama/blob/main/transformer_cu/mha.cpp): `transformer_cu/mha.cpp`
- [DDR Weights](https://github.com/mivey/hls_llama/blob/main/transformer_cu/transformer_kernel.cpp): `transformer_cu/transformer_kernel.cpp`

### Reference Inference

- [Int8 C Reference](https://github.com/mivey/hls_llama/blob/main/llama2_software/runq.c): `llama2_software/runq.c`

### Other components

- **Prompt User**
- **Versal VE2302**

## Connections

- Prompt User → Generation Driver: provides prompt
- Generation Driver → BPE Tokenizer: tokenizes
- Generation Driver → Checkpoint Repacker: loads checkpoint
- Generation Driver → Prefill Decode Loop: starts generation
- Prefill Decode Loop → IP / XRT Engines: drives steps
- IP / XRT Engines → Versal VE2302: starts kernel
- Checkpoint Repacker → Versal VE2302: loads buffers
- Versal VE2302 → Transformer Kernel: executes
- Transformer Kernel → 49-Step FSM: builds schedule
- Transformer Kernel → Weight Prefetch: streams weights
- 49-Step FSM → State Selector: dispatches states
- Weight Prefetch → Split GeMV: feeds weights
- State Selector → RMSNorm Residual: invokes
- State Selector → RoPE Attention: invokes
- State Selector → SwiGLU: invokes
- State Selector → Int8 Quantizer: invokes
- Int8 Quantizer → Split GeMV: feeds activations
- Split GeMV → Top-64 Sampler: streams logits
- Split GeMV → Token Buffer: writes activations
- Top-64 Sampler → Token Buffer: writes sampled token
- RoPE Attention → KV Cache: reads and updates
- DDR Weights → Weight Prefetch: reads weights
- Token Buffer → Transformer Kernel: reads token
- Transformer Kernel → IP / XRT Engines: returns token
- IP / XRT Engines → Prefill Decode Loop: returns token
- Prefill Decode Loop → Generation Driver: emits decoded output
- BPE Tokenizer → Generation Driver: decodes token
- Int8 C Reference → Generation Driver: reference behavior