How mivey/hls_llama works
HLS_llama is a Versal VE2302 accelerator and A72 host for int8 LLaMA-2 inference. The user supplies a prompt and checkpoint; the host tokenizes the prompt, prepares model weights and drives prefill/decode steps through either of two engines. A single programmable-logic kernel schedules transformer stages, updates the KV cache and samples tokens, which the host decodes for output. The README documents the reference C inference path and major kernel optimizations; implementation links for some host workflow details are inferred from the documented module roles, since their source was not sampled.
Main entry points: fastforward.cpp Generation Driver · transformer_kernel.cpp State Selector · generate_loop.h Prefill Decode Loop · compute_units.h IP / XRT Engines
Components 21
Host Inference
- Generation Driver
accelerated_llama/fastforward.cpp - BPE Tokenizer
accelerated_llama/fastforward.cpp - Checkpoint Repacker
accelerated_llama/compute_units.h - Prefill Decode Loop
accelerated_llama/generate_loop.h - IP / XRT Engines
accelerated_llama/compute_units.h
Transformer Kernel
- Transformer Kernel
transformer_cu/transformer_kernel.cpp - 49-Step FSM
transformer_cu/transformer_kernel.cpp - Weight Prefetch
transformer_cu/transformer_kernel.cpp - State Selector
transformer_cu/transformer_kernel.cpp
Compute Stages
- RMSNorm Residual
transformer_cu/rmsnorm.cpp - RoPE Attention
transformer_cu/mha.cpp - Int8 Quantizer
transformer_cu/quantizer.cpp - Split GeMV
transformer_cu/matmult.cpp - SwiGLU
transformer_cu/swiglu_kernel.cpp - Top-64 Sampler
transformer_cu/combiner.h
Model State
- Token Buffer
transformer_cu/transformer_kernel.cpp - KV Cache
transformer_cu/mha.cpp - DDR Weights
transformer_cu/transformer_kernel.cpp
Reference Inference
- Int8 C Reference
llama2_software/runq.c
Other components
- Prompt User
- Versal VE2302
Connections 28
- Prompt User → Generation Driver: provides prompt
- Generation Driver → BPE Tokenizer: tokenizes
- Generation Driver → Checkpoint Repacker: loads checkpoint
- Generation Driver → Prefill Decode Loop: starts generation
- Prefill Decode Loop → IP / XRT Engines: drives steps
- IP / XRT Engines → Versal VE2302: starts kernel
- Checkpoint Repacker → Versal VE2302: loads buffers
- Versal VE2302 → Transformer Kernel: executes
- Transformer Kernel → 49-Step FSM: builds schedule
- Transformer Kernel → Weight Prefetch: streams weights
- 49-Step FSM → State Selector: dispatches states
- Weight Prefetch → Split GeMV: feeds weights
- State Selector → RMSNorm Residual: invokes
- State Selector → RoPE Attention: invokes
- State Selector → SwiGLU: invokes
- State Selector → Int8 Quantizer: invokes
- Int8 Quantizer → Split GeMV: feeds activations
- Split GeMV → Top-64 Sampler: streams logits
- Split GeMV → Token Buffer: writes activations
- Top-64 Sampler → Token Buffer: writes sampled token
- RoPE Attention → KV Cache: reads and updates
- DDR Weights → Weight Prefetch: reads weights
- Token Buffer → Transformer Kernel: reads token
- Transformer Kernel → IP / XRT Engines: returns token
- IP / XRT Engines → Prefill Decode Loop: returns token
- Prefill Decode Loop → Generation Driver: emits decoded output
- BPE Tokenizer → Generation Driver: decodes token
- Int8 C Reference → Generation Driver: reference behavior