# talystalys/abra architecture

> Architecture diagram of the GitHub repository talystalys/abra, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/talystalys/abra
- Repository: https://github.com/talystalys/abra

## Overview

Abra is presented as a Pascal-focused LLM inference engine: a user submits a prompt to the Qwen3 runner, which produces generated tokens using custom quantized GPU kernels and optional dual-GPU execution. The README also describes kernel autotuning, vocabulary splitting, and activation reuse; the tree lists supporting lookup and qualification tools. The supplied excerpts do not expose the kernel implementation or most runtime wiring, so those subsystems are represented at the documented level, with evidence limits reflected in the edges.

Main entry points: [`tools`](https://github.com/talystalys/abra/tree/main/tools) (Inference engine), [`kernels`](https://github.com/talystalys/abra/tree/main/kernels) (Q4_K × Q8_1 family), [`tools/qwen3-4b.py`](https://github.com/talystalys/abra/blob/main/tools/qwen3-4b.py) (Qwen3 runner), [`tools/qwen3-4b-lookup.py`](https://github.com/talystalys/abra/blob/main/tools/qwen3-4b-lookup.py) (Prompt lookup)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Inference runtime"]
  node_runner["Qwen3 runner<br/>[qwen3-4b.py]"]
  node_lookup["Prompt lookup<br/>[qwen3-4b-lookup.py]"]
  node_engine["Inference engine"]
  node_tensor["Tensor parallelism"]
  node_vocab["Vocabulary row split"]
  node_reuse["Q8 activation reuse"]
end

subgraph group_compute["GPU compute"]
  node_dp4a["DP4A microkernels"]
  node_q4k["Q4_K × Q8_1 family"]
  node_autotune["Kernel autotuning<br/>[run-m4-autotune.py]"]
end

subgraph group_qualification["Qualification tools"]
  node_lookup_eval["Lookup evaluation"]
  node_attention_eval["Attention qualification"]
  node_port_bench["Port benchmarking<br/>[port-benchmark.py]"]
  node_runtime_compare["Runtime comparison"]
end

node_user(("Inference user"))
node_gpu["Pascal GPUs"]
node_nccl["NCCL"]

node_user -->|"submits prompt"| node_runner
node_runner -->|"runs inference"| node_engine
node_engine -.->|"executes on"| node_gpu
node_engine -->|"uses kernels"| node_q4k
node_q4k -->|"uses DP4A"| node_dp4a
node_autotune -->|"tunes variants"| node_q4k
node_engine -->|"supports splitting"| node_tensor
node_tensor -->|"communicates via"| node_nccl
node_engine -->|"splits projection"| node_vocab
node_engine -->|"reuses activations"| node_reuse
node_user -.->|"requests lookup"| node_lookup
node_lookup_eval -.->|"evaluates"| node_lookup
node_attention_eval -.->|"qualifies attention"| node_engine
node_port_bench -.->|"benchmarks port"| node_engine
node_runtime_compare -.->|"compares runtime"| node_engine

click node_runner "https://github.com/talystalys/abra/blob/main/tools/qwen3-4b.py"
click node_lookup "https://github.com/talystalys/abra/blob/main/tools/qwen3-4b-lookup.py"
click node_engine "https://github.com/talystalys/abra/tree/main/tools"
click node_dp4a "https://github.com/talystalys/abra/tree/main/kernels"
click node_q4k "https://github.com/talystalys/abra/tree/main/kernels"
click node_autotune "https://github.com/talystalys/abra/blob/main/tools/run-m4-autotune.py"
click node_tensor "https://github.com/talystalys/abra/tree/main/tools"
click node_vocab "https://github.com/talystalys/abra/tree/main/tools"
click node_reuse "https://github.com/talystalys/abra/tree/main/tools"
click node_lookup_eval "https://github.com/talystalys/abra/blob/main/tools/lookup-experiments.py"
click node_attention_eval "https://github.com/talystalys/abra/blob/main/tools/qualify-flash-attention.py"
click node_port_bench "https://github.com/talystalys/abra/blob/main/tools/port-benchmark.py"
click node_runtime_compare "https://github.com/talystalys/abra/blob/main/tools/compare-runtimes.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner,node_lookup,node_engine,node_tensor,node_vocab,node_reuse,node_user toneBlue
class node_dp4a,node_q4k,node_autotune toneAmber
class node_lookup_eval,node_attention_eval,node_port_bench,node_runtime_compare toneMint
class node_gpu,node_nccl toneIndigo
```

## Components

### Inference runtime

- [Qwen3 runner](https://github.com/talystalys/abra/blob/main/tools/qwen3-4b.py): `tools/qwen3-4b.py`
- [Prompt lookup](https://github.com/talystalys/abra/blob/main/tools/qwen3-4b-lookup.py): `tools/qwen3-4b-lookup.py`
- [Inference engine](https://github.com/talystalys/abra/tree/main/tools): `tools`
- [Tensor parallelism](https://github.com/talystalys/abra/tree/main/tools): `tools`
- [Vocabulary row split](https://github.com/talystalys/abra/tree/main/tools): `tools`
- [Q8 activation reuse](https://github.com/talystalys/abra/tree/main/tools): `tools`

### GPU compute

- [DP4A microkernels](https://github.com/talystalys/abra/tree/main/kernels): `kernels`
- [Q4_K × Q8_1 family](https://github.com/talystalys/abra/tree/main/kernels): `kernels`
- [Kernel autotuning](https://github.com/talystalys/abra/blob/main/tools/run-m4-autotune.py): `tools/run-m4-autotune.py`

### Qualification tools

- [Lookup evaluation](https://github.com/talystalys/abra/blob/main/tools/lookup-experiments.py): `tools/lookup-experiments.py`
- [Attention qualification](https://github.com/talystalys/abra/blob/main/tools/qualify-flash-attention.py): `tools/qualify-flash-attention.py`
- [Port benchmarking](https://github.com/talystalys/abra/blob/main/tools/port-benchmark.py): `tools/port-benchmark.py`
- [Runtime comparison](https://github.com/talystalys/abra/blob/main/tools/compare-runtimes.py): `tools/compare-runtimes.py`

### Other components

- **Inference user**
- **Pascal GPUs**
- **NCCL**

## Connections

- Inference user → Qwen3 runner: submits prompt
- Qwen3 runner → Inference engine: runs inference
- Inference engine → Pascal GPUs: executes on
- Inference engine → Q4_K × Q8_1 family: uses kernels
- Q4_K × Q8_1 family → DP4A microkernels: uses DP4A
- Kernel autotuning → Q4_K × Q8_1 family: tunes variants
- Inference engine → Tensor parallelism: supports splitting
- Tensor parallelism → NCCL: communicates via
- Inference engine → Vocabulary row split: splits projection
- Inference engine → Q8 activation reuse: reuses activations
- Inference user → Prompt lookup: requests lookup
- Lookup evaluation → Prompt lookup: evaluates
- Attention qualification → Inference engine: qualifies attention
- Port benchmarking → Inference engine: benchmarks port
- Runtime comparison → Inference engine: compares runtime