How talystalys/abra works
Abra is presented as a Pascal-focused LLM inference engine: a user submits a prompt to the Qwen3 runner, which produces generated tokens using custom quantized GPU kernels and optional dual-GPU execution. The README also describes kernel autotuning, vocabulary splitting, and activation reuse; the tree lists supporting lookup and qualification tools. The supplied excerpts do not expose the kernel implementation or most runtime wiring, so those subsystems are represented at the documented level, with evidence limits reflected in the edges.
Main entry points: tools Inference engine · kernels Q4_K × Q8_1 family · qwen3-4b.py Qwen3 runner · qwen3-4b-lookup.py Prompt lookup
Components 16
Inference runtime
- Qwen3 runner
tools/qwen3-4b.py - Prompt lookup
tools/qwen3-4b-lookup.py - Inference engine
tools - Tensor parallelism
tools - Vocabulary row split
tools - Q8 activation reuse
tools
GPU compute
- DP4A microkernels
kernels - Q4_K × Q8_1 family
kernels - Kernel autotuning
tools/run-m4-autotune.py
Qualification tools
- Lookup evaluation
tools/lookup-experiments.py - Attention qualification
tools/qualify-flash-attention.py - Port benchmarking
tools/port-benchmark.py - Runtime comparison
tools/compare-runtimes.py
Other components
- Inference user
- Pascal GPUs
- NCCL
Connections 15
- Inference user → Qwen3 runner: submits prompt
- Qwen3 runner → Inference engine: runs inference
- Inference engine → Pascal GPUs: executes on
- Inference engine → Q4_K × Q8_1 family: uses kernels
- Q4_K × Q8_1 family → DP4A microkernels: uses DP4A
- Kernel autotuning → Q4_K × Q8_1 family: tunes variants
- Inference engine → Tensor parallelism: supports splitting
- Tensor parallelism → NCCL: communicates via
- Inference engine → Vocabulary row split: splits projection
- Inference engine → Q8 activation reuse: reuses activations
- Inference user → Prompt lookup: requests lookup
- Lookup evaluation → Prompt lookup: evaluates
- Attention qualification → Inference engine: qualifies attention
- Port benchmarking → Inference engine: benchmarks port
- Runtime comparison → Inference engine: compares runtime