How openjiuwen-ai/agent-infer works
AgentInfer combines agent-aware inference serving and scheduling with AgentBench, which evaluates serving engines using live agent runs or trace replay. The primary serving path intercepts an opted-in vLLM serve command, builds the AgentCache integration, normalizes request identity, and schedules work through a shared runtime and policy. Separate branches provide benchmark proxying and analysis, plus router session affinity. Edges are limited to README statements and source references in the provided excerpts and SOURCE INDEX; unsampled subsystem internals are represented without guessed call chains.
Main entry points: runtime.py Program scheduler · runner.py Benchmark runner · controller.py Cache controller · main.py vLLM CLI hook
Components 21
Serving entrypoints
- vLLM CLI hook
agentinfer/agentcache/entrypoints/cli/main.py - Serve takeover
agentinfer/agentcache/entrypoints/cli/serve_profile.py - Cache controller
agentinfer/agentcache/core/controller.py - Cache scheduler
agentinfer/agentcache/core/scheduler.py - vLLM adapter
agentinfer/agentcache/core/api_adapter.py
Scheduling runtime
- Request identity
agentinfer/scheduling/identity.py - Program scheduler
agentinfer/scheduling/runtime.py - Progress TTL policy
agentinfer/scheduling/progress_ttl/strategy.py - Scheduling contracts
agentinfer/scheduling - Backend facts
agentinfer/scheduling/backend.py
AgentBench
- Benchmark runner
agentinfer/agentbench/benchkit/runner.py - Agent runners
agentinfer/agentbench/agents - Trace replay
agentinfer/agentbench/replay - Request proxy
agentinfer/agentbench/request_proxy/server.py - Request traces
agentinfer/agentbench/request_proxy/request_trace.py - Session analysis
agentinfer/agentbench/benchkit/session_analysis.py - Metrics collection
agentinfer/agentbench/benchkit/metrics
Router affinity
- Affinity guest
agentinfer/agentrouter/agent_hint_affinity/src/lib.rs
Other components
- Serving operator
- vLLM engine
- vLLM Router
Connections 17
- Serving operator → vLLM CLI hook: invokes
README.md - vLLM CLI hook → Serve takeover: routes serve
agentinfer/agentcache/entrypoints/cli/main.py - Serve takeover → vLLM engine: activates serving
README.md - Cache controller → Program scheduler: builds scheduler
agentinfer/agentcache/core/controller.py - Cache controller → Progress TTL policy: builds policy
agentinfer/agentcache/core/controller.py - Cache scheduler → Program scheduler: adapts scheduling
- vLLM adapter → Program scheduler: adapts requests
- Program scheduler → Request identity: uses identity
agentinfer/scheduling/runtime.py - Program scheduler → Progress TTL policy: dispatches decisions
agentinfer/scheduling/runtime.py - Program scheduler → Scheduling contracts: uses contracts
agentinfer/scheduling/runtime.py - Program scheduler → Backend facts: reads backend facts
agentinfer/scheduling/runtime.py - Benchmark runner → Agent runners: runs agents
README.md - Benchmark runner → Trace replay: replays traces
README.md - Request proxy → Request traces: records requests
agentinfer/agentbench/request_proxy/server.py - Session analysis → Request traces: reads facts
agentinfer/agentbench/benchkit/session_analysis.py - Metrics collection → Benchmark runner: collects metrics
- Affinity guest → vLLM Router: adds affinity
README.md