# kush-singh-26/inference_engine architecture

> Architecture diagram of the GitHub repository kush-singh-26/inference_engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kush-singh-26/inference_engine
- Repository: https://github.com/kush-singh-26/inference_engine

## Overview

This repository is a CPU C++ Transformer inference engine for Hindi-English translation, with a Python-based model preparation path. The user workflow accepts text, tokenizes it, runs Transformer inference, and emits generated text; benchmark mode reuses that engine with fixed prompts. The README also documents BPE assets, hybrid Int8 quantization, preallocated workspaces, AVX2/OpenMP kernels, and KV caching. The excerpts show the CLI invoking tokenization and encoding, plus model-to-tensor operations; decoder details and some runtime connections are not fully excerpted, so those links are omitted or documented at the engine level.

Main entry points: [`main.cpp`](https://github.com/kush-singh-26/inference_engine/blob/main/main.cpp) (Inference CLI), [`models/model_q8.bin`](https://github.com/kush-singh-26/inference_engine/blob/main/models/model_q8.bin) (Quantized model), [`scripts/export_model.py`](https://github.com/kush-singh-26/inference_engine/blob/main/scripts/export_model.py) (Weight exporter), [`scripts/quantize.py`](https://github.com/kush-singh-26/inference_engine/blob/main/scripts/quantize.py) (Int8 quantizer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["C++ inference"]
  node_cli["Inference CLI<br/>[main.cpp]"]
  node_bpe["BPE tokenizer<br/>[tokenizer.h]"]
  node_transformer["Transformer engine<br/>[model.cpp]"]
  node_weights[("Quantized model<br/>[model_q8.bin]")]
  node_vocab["Vocabulary<br/>[vocab.txt]"]
  node_merges["BPE merge rules<br/>[merges.txt]"]
  node_model_api["Model API<br/>[model.h]"]
end

subgraph group_math["Tensor kernels"]
  node_tensor["Tensor operations<br/>[tensor.cpp]"]
  node_tensor_types["Tensor types<br/>[tensor.h]"]
end

subgraph group_prep["Model preparation"]
  node_pytorch_model["Reference Transformer<br/>[inference.py]"]
  node_optimized_model["Optimized reference<br/>[inference_opt.py]"]
  node_export_model["Weight exporter<br/>[export_model.py]"]
  node_quantize["Int8 quantizer<br/>[quantize.py]"]
  node_export_tokenizer["Tokenizer exporter"]
end

node_user(("Text user"))

node_user -->|"provides text"| node_cli
node_cli -->|"tokenizes input"| node_bpe
node_cli -->|"runs inference"| node_transformer
node_cli -->|"loads weights"| node_weights
node_cli -->|"loads vocabulary"| node_vocab
node_cli -->|"loads merges"| node_merges
node_transformer -->|"uses kernels"| node_tensor
node_model_api -->|"declares tensor use"| node_tensor_types
node_export_model -->|"imports model"| node_pytorch_model
node_export_model -->|"exports weights"| node_weights
node_quantize -->|"imports model"| node_optimized_model
node_quantize -->|"writes Q8 model"| node_weights
node_export_tokenizer -->|"exports merge rules"| node_merges

click node_cli "https://github.com/kush-singh-26/inference_engine/blob/main/main.cpp"
click node_bpe "https://github.com/kush-singh-26/inference_engine/blob/main/include/tokenizer.h"
click node_transformer "https://github.com/kush-singh-26/inference_engine/blob/main/src/model.cpp"
click node_weights "https://github.com/kush-singh-26/inference_engine/blob/main/models/model_q8.bin"
click node_vocab "https://github.com/kush-singh-26/inference_engine/blob/main/models/vocab.txt"
click node_merges "https://github.com/kush-singh-26/inference_engine/blob/main/models/merges.txt"
click node_tensor "https://github.com/kush-singh-26/inference_engine/blob/main/src/tensor.cpp"
click node_tensor_types "https://github.com/kush-singh-26/inference_engine/blob/main/include/tensor.h"
click node_model_api "https://github.com/kush-singh-26/inference_engine/blob/main/include/model.h"
click node_pytorch_model "https://github.com/kush-singh-26/inference_engine/blob/main/scripts/inference.py"
click node_optimized_model "https://github.com/kush-singh-26/inference_engine/blob/main/scripts/inference_opt.py"
click node_export_model "https://github.com/kush-singh-26/inference_engine/blob/main/scripts/export_model.py"
click node_quantize "https://github.com/kush-singh-26/inference_engine/blob/main/scripts/quantize.py"
click node_export_tokenizer "https://github.com/kush-singh-26/inference_engine/blob/main/scripts/export_tokenizer.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_bpe,node_transformer,node_weights,node_vocab,node_merges,node_model_api,node_user toneBlue
class node_tensor,node_tensor_types toneAmber
class node_pytorch_model,node_optimized_model,node_export_model,node_quantize,node_export_tokenizer toneMint
```

## Components

### C++ inference

- [Inference CLI](https://github.com/kush-singh-26/inference_engine/blob/main/main.cpp): `main.cpp`
- [BPE tokenizer](https://github.com/kush-singh-26/inference_engine/blob/main/include/tokenizer.h): `include/tokenizer.h`
- [Transformer engine](https://github.com/kush-singh-26/inference_engine/blob/main/src/model.cpp): `src/model.cpp`
- [Quantized model](https://github.com/kush-singh-26/inference_engine/blob/main/models/model_q8.bin): `models/model_q8.bin`
- [Vocabulary](https://github.com/kush-singh-26/inference_engine/blob/main/models/vocab.txt): `models/vocab.txt`
- [BPE merge rules](https://github.com/kush-singh-26/inference_engine/blob/main/models/merges.txt): `models/merges.txt`
- [Model API](https://github.com/kush-singh-26/inference_engine/blob/main/include/model.h): `include/model.h`

### Tensor kernels

- [Tensor operations](https://github.com/kush-singh-26/inference_engine/blob/main/src/tensor.cpp): `src/tensor.cpp`
- [Tensor types](https://github.com/kush-singh-26/inference_engine/blob/main/include/tensor.h): `include/tensor.h`

### Model preparation

- [Reference Transformer](https://github.com/kush-singh-26/inference_engine/blob/main/scripts/inference.py): `scripts/inference.py`
- [Optimized reference](https://github.com/kush-singh-26/inference_engine/blob/main/scripts/inference_opt.py): `scripts/inference_opt.py`
- [Weight exporter](https://github.com/kush-singh-26/inference_engine/blob/main/scripts/export_model.py): `scripts/export_model.py`
- [Int8 quantizer](https://github.com/kush-singh-26/inference_engine/blob/main/scripts/quantize.py): `scripts/quantize.py`
- [Tokenizer exporter](https://github.com/kush-singh-26/inference_engine/blob/main/scripts/export_tokenizer.py): `scripts/export_tokenizer.py`

### Other components

- **Text user**

## Connections

- Text user → Inference CLI: provides text (evidence: `README.md`)
- Inference CLI → BPE tokenizer: tokenizes input (evidence: `main.cpp`)
- Inference CLI → Transformer engine: runs inference (evidence: `README.md`)
- Inference CLI → Quantized model: loads weights (evidence: `main.cpp`)
- Inference CLI → Vocabulary: loads vocabulary (evidence: `main.cpp`)
- Inference CLI → BPE merge rules: loads merges (evidence: `main.cpp`)
- Transformer engine → Tensor operations: uses kernels (evidence: `src/model.cpp`)
- Model API → Tensor types: declares tensor use (evidence: `include/model.h`)
- Weight exporter → Reference Transformer: imports model (evidence: `scripts/export_model.py`)
- Weight exporter → Quantized model: exports weights (evidence: `scripts/export_model.py`)
- Int8 quantizer → Optimized reference: imports model (evidence: `scripts/quantize.py`)
- Int8 quantizer → Quantized model: writes Q8 model (evidence: `scripts/quantize.py`)
- Tokenizer exporter → BPE merge rules: exports merge rules (evidence: `scripts/export_tokenizer.py`)
