How infervisor/plow works
Plow is an ahead-of-time inference engine: a model build is compiled into a packet and weights, loaded by the runtime, then served through an OpenAI-compatible API. The graph preserves compiler/packet stages, asset loading, request serving, scheduling, execution backends, and KV prefix sharing. The README documents the end-to-end workflow and speech capabilities; source-index references and excerpts support selected internal links. Other unsampled runtime subsystems are represented at directory level, with uncertain wiring omitted.
Main entry points: engine.rs Serving engine · packet_runtime.rs Packet runtime · manager.rs Model manager · main.rs Model compiler
Components 20
Model compilation
- Model compiler
crates/plowc/src/main.rs - Model graphs
crates/nn-graph/src/models - Packet generation
crates/devgen/src/pipeline.rs - Packet builder
crates/packet/src/devbuild.rs
Assets and loading
- Asset contracts
crates/plow-asset/src/packet_pipeline.rs - Packet image
crates/plowrt/src/asset/packet_image.rs - Device blob reader
crates/plowrt/src/asset/devblob.rs - Serve assets
crates/plowrt/src/asset/serve.rs
Serving and scheduling
- Model manager
crates/plowrt/src/serve/manager.rs - HTTP API
crates/plowrt/src/serve/mux.rs - Serving engine
crates/plowrt/src/serve/engine.rs - Request scheduler
crates/plowrt/src/sched
Execution and memory
- Packet runtime
crates/plowrt/src/exec/packet_runtime.rs - GPU execution
crates/plowrt/src/exec/gpu.rs - CPU execution
crates/plowrt/src/exec/cpu/engine.rs - AMD device backend
crates/plowrt/src/device/hsa.rs - KV prefix sharing
crates/plowrt/src/memory/vmm.rs
Speech capabilities
- Speech recognition
crates/plowrt/src/asr - Speech generation
crates/plowrt/src/tts
Other components
- Client / operator
Connections 19
- Client / operator → Model compiler: builds model
README.md - Model compiler → Packet builder: emits packet
README.md - Packet builder → Asset contracts: packages assets
- Client / operator → Model manager: loads model
README.md - Model manager → Packet image: opens packet
crates/plowrt/src/serve/manager.rs - Model manager → Device blob reader: reads device blob
crates/plowrt/src/serve/manager.rs - Model manager → Serve assets: loads serve settings
crates/plowrt/src/serve/manager.rs - Client / operator → HTTP API: sends request
README.md - HTTP API → Serving engine: routes requests
- Serving engine → Request scheduler: schedules work
crates/plowrt/src/serve/engine.rs - Serving engine → GPU execution: dispatches GPU work
crates/plowrt/src/serve/engine.rs - Serving engine → AMD device backend: uses AMD backend
crates/plowrt/src/serve/engine.rs - Packet runtime → Packet image: consumes packet
crates/plowrt/src/exec/packet_runtime.rs - Packet runtime → Device blob reader: reads blob data
crates/plowrt/src/exec/packet_runtime.rs - Packet runtime → GPU execution: dispatches GPU ops
crates/plowrt/src/exec/packet_runtime.rs - Packet runtime → CPU execution: uses CPU engine
crates/plowrt/src/exec/packet_runtime.rs - Serving engine → KV prefix sharing: uses KV sharing
crates/plowrt/src/serve/engine.rs - Client / operator → Speech recognition: requests transcription
README.md - Client / operator → Speech generation: requests speech
README.md