How brave30/llm-inference-engine works
This repository is a GPU LLM inference runtime. Its primary workflow accepts tokenized prompts, schedules mixed decode and prefill work, allocates paged KV-cache space, runs Llama forward passes and attention kernels, then samples and returns generated tokens. The graph also retains weight/config loading and the major kernel and CUDA-graph paths. Evidence is limited to the README and indexed excerpts; unsampled implementation details are omitted where relationships are not documented.
Main entry points: engine.py Inference engine · model.py Llama decoder · scheduler.py Batch scheduler · block_manager.py KV page allocator
Components 16
Request API
- Inference engine
inference_engine/engine.py - Tokenizer
Batch orchestration
- Request sequence
inference_engine/sequence.py - Batch scheduler
inference_engine/scheduler.py - KV page allocator
inference_engine/block_manager.py
Model execution
- Llama decoder
inference_engine/model.py - Paged KV cache
inference_engine/model.py - Fused RMSNorm
inference_engine/kernels/rmsnorm.py - Paged decode attention
inference_engine/kernels/decode_attention.py - Prefill SDPA
inference_engine/model.py - Token sampling
inference_engine/sampling.py
Runtime resources
- Engine configuration
inference_engine/config.py - Model weights
inference_engine/weights.py - Model repository
- CUDA GPU
Other components
- Caller
Connections 18
- Caller → Inference engine: submits prompts
README.md - Inference engine → Tokenizer: encodes prompts
README.md - Inference engine → Request sequence: creates sequences
inference_engine/engine.py - Inference engine → Batch scheduler: schedules work
inference_engine/engine.py - Batch scheduler → KV page allocator: reserves pages
inference_engine/scheduler.py - Inference engine → Llama decoder: runs forward pass
inference_engine/engine.py - Llama decoder → Paged KV cache: writes and reads
inference_engine/model.py - Llama decoder → Fused RMSNorm: normalizes
inference_engine/model.py - Llama decoder → Paged decode attention: dispatches decode
inference_engine/model.py - Llama decoder → Prefill SDPA: runs prefill
inference_engine/model.py - Inference engine → Token sampling: samples logits
inference_engine/engine.py - Inference engine → Engine configuration: reads settings
inference_engine/engine.py - Inference engine → Model weights: loads weights
inference_engine/engine.py - Model weights → Model repository: downloads model
inference_engine/weights.py - Inference engine → Tokenizer: loads tokenizer
inference_engine/engine.py - Inference engine → CUDA GPU: executes on
README.md - Inference engine → Caller: returns outputs
README.md - KV page allocator → Request sequence: updates block table
inference_engine/block_manager.py