How w2315516642/mini-vllm works
MiniVLLM is a Python LLM inference runtime: callers submit prompts and sampling settings, then receive generated text and token-level results. The engine schedules prompt and decode work, tracks sequence state and KV blocks, dispatches workers for model execution, and collects outputs. The sampled code supports prefix-aware KV caching and tensor-parallel LLaMA inference; the tree also shows Ray utilities for distributed execution. The README is empty, and unsampled subsystems are included only where their paths and roles are evident from the tree.
Main entry points: llm_engine.py LLM Engine · llm.py LLM API · scheduler.py Scheduler · model_runner.py Model Runner
Components 20
Public API
- LLM API
minivllm/entrypoints/llm.py - Tokenizer
minivllm/engine/tokenizer_utils.py - Sampling Params
minivllm/sampling_params.py - Request Outputs
minivllm/outputs.py
Engine Runtime
- LLM Engine
minivllm/engine/llm_engine.py - Scheduler
minivllm/kv_cache/scheduler.py - Worker
minivllm/worker/worker.py - Ray Runtime
minivllm/engine/ray_utils.py
Sequence & Cache
- Sequences
minivllm/sequence.py - Block Manager
minivllm/kv_cache/block_manager.py - Scheduling Policy
minivllm/kv_cache/policy.py - KV Cache Engine
minivllm/worker/cache_engine.py
Model Inference
- Model Runner
minivllm/engine/model_runner.py - Input Metadata
minivllm/model_executor/input_metadata.py - Model Loader
minivllm/model_executor/model_loader.py - LLaMA Model
minivllm/model_executor/models/llama.py - Token Sampler
minivllm/model_executor/layers/sampler.py - Attention Layer
minivllm/model_executor/layers/attention.py - Attention Kernels
csrc/attention.cpp
Other components
- Application
Connections 22
- Application → LLM API: submits prompts
- LLM API → LLM Engine: adds requests
- LLM API → LLM Engine: steps engine
- LLM API → Tokenizer: encodes prompts
- LLM API → Tokenizer: decodes results
- LLM API → Sampling Params: sets sampling
- LLM Engine → Scheduler: schedules work
- LLM Engine → Sequences: creates sequences
- LLM Engine → Worker: dispatches execution
- LLM Engine → Request Outputs: builds results
- Scheduler → Sequences: tracks sequence state
- Scheduler → Block Manager: allocates blocks
- Scheduler → Scheduling Policy: selects work
- Model Runner → Input Metadata: builds inputs
- Worker → Model Runner: runs model
- Worker → KV Cache Engine: manages cache
- Model Runner → Model Loader: loads model
- Model Runner → LLaMA Model: executes forward
- LLaMA Model → Attention Layer: computes attention
- LLaMA Model → Token Sampler: samples tokens
- Attention Layer → Attention Kernels: invokes kernels
- LLM Engine → Ray Runtime: uses distributed workers