# openjiuwen-ai/agent-infer architecture

> Architecture diagram of the GitHub repository openjiuwen-ai/agent-infer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openjiuwen-ai/agent-infer
- Repository: https://github.com/openjiuwen-ai/agent-infer

## Overview

AgentInfer combines agent-aware inference serving and scheduling with AgentBench, which evaluates serving engines using live agent runs or trace replay. The primary serving path intercepts an opted-in vLLM serve command, builds the AgentCache integration, normalizes request identity, and schedules work through a shared runtime and policy. Separate branches provide benchmark proxying and analysis, plus router session affinity. Edges are limited to README statements and source references in the provided excerpts and SOURCE INDEX; unsampled subsystem internals are represented without guessed call chains.

Main entry points: [`agentinfer/scheduling/runtime.py`](https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/scheduling/runtime.py) (Program scheduler), [`agentinfer/agentbench/benchkit/runner.py`](https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentbench/benchkit/runner.py) (Benchmark runner), [`agentinfer/agentcache/core/controller.py`](https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentcache/core/controller.py) (Cache controller), [`agentinfer/agentcache/entrypoints/cli/main.py`](https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentcache/entrypoints/cli/main.py) (vLLM CLI hook)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Serving entrypoints"]
  node_vllm_cli["vLLM CLI hook<br/>[main.py]"]
  node_serve_profile["Serve takeover<br/>[serve_profile.py]"]
  node_cache_controller["Cache controller<br/>[controller.py]"]
  node_cache_scheduler["Cache scheduler<br/>[scheduler.py]"]
  node_api_adapter["vLLM adapter<br/>[api_adapter.py]"]
end

subgraph group_scheduler["Scheduling runtime"]
  node_identity["Request identity<br/>[identity.py]"]
  node_runtime["Program scheduler<br/>[runtime.py]"]
  node_policy["Progress TTL policy<br/>[strategy.py]"]
  node_schedule_contracts["Scheduling contracts"]
  node_backend["Backend facts<br/>[backend.py]"]
end

subgraph group_bench["AgentBench"]
  node_agentbench["Benchmark runner<br/>[runner.py]"]
  node_agent_runners["Agent runners"]
  node_trace_replay["Trace replay"]
  node_request_proxy["Request proxy<br/>[server.py]"]
  node_request_trace["Request traces<br/>[request_trace.py]"]
  node_bench_analysis["Session analysis"]
  node_bench_metrics["Metrics collection"]
end

subgraph group_router["Router affinity"]
  node_affinity_guest["Affinity guest<br/>[lib.rs]"]
end

node_user(("Serving operator"))
node_vllm_engine["vLLM engine"]
node_router_service["vLLM Router"]

node_user -->|"invokes"| node_vllm_cli
node_vllm_cli -->|"routes serve"| node_serve_profile
node_serve_profile -->|"activates serving"| node_vllm_engine
node_cache_controller -->|"builds scheduler"| node_runtime
node_cache_controller -->|"builds policy"| node_policy
node_cache_scheduler -.->|"adapts scheduling"| node_runtime
node_api_adapter -.->|"adapts requests"| node_runtime
node_runtime -->|"uses identity"| node_identity
node_runtime -->|"dispatches decisions"| node_policy
node_runtime -->|"uses contracts"| node_schedule_contracts
node_runtime -->|"reads backend facts"| node_backend
node_agentbench -.->|"runs agents"| node_agent_runners
node_agentbench -.->|"replays traces"| node_trace_replay
node_request_proxy -->|"records requests"| node_request_trace
node_bench_analysis -->|"reads facts"| node_request_trace
node_bench_metrics -.->|"collects metrics"| node_agentbench
node_affinity_guest -->|"adds affinity"| node_router_service

click node_vllm_cli "https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentcache/entrypoints/cli/main.py"
click node_serve_profile "https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentcache/entrypoints/cli/serve_profile.py"
click node_cache_controller "https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentcache/core/controller.py"
click node_cache_scheduler "https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentcache/core/scheduler.py"
click node_api_adapter "https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentcache/core/api_adapter.py"
click node_identity "https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/scheduling/identity.py"
click node_runtime "https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/scheduling/runtime.py"
click node_policy "https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/scheduling/progress_ttl/strategy.py"
click node_schedule_contracts "https://github.com/openjiuwen-ai/agent-infer/tree/main/agentinfer/scheduling"
click node_backend "https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/scheduling/backend.py"
click node_agentbench "https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentbench/benchkit/runner.py"
click node_agent_runners "https://github.com/openjiuwen-ai/agent-infer/tree/main/agentinfer/agentbench/agents"
click node_trace_replay "https://github.com/openjiuwen-ai/agent-infer/tree/main/agentinfer/agentbench/replay"
click node_request_proxy "https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentbench/request_proxy/server.py"
click node_request_trace "https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentbench/request_proxy/request_trace.py"
click node_bench_analysis "https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentbench/benchkit/session_analysis.py"
click node_bench_metrics "https://github.com/openjiuwen-ai/agent-infer/tree/main/agentinfer/agentbench/benchkit/metrics"
click node_affinity_guest "https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentrouter/agent_hint_affinity/src/lib.rs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_vllm_cli,node_serve_profile,node_cache_controller,node_cache_scheduler,node_api_adapter toneBlue
class node_identity,node_runtime,node_policy,node_schedule_contracts,node_backend toneAmber
class node_agentbench,node_agent_runners,node_trace_replay,node_request_proxy,node_request_trace,node_bench_analysis,node_bench_metrics,node_router_service toneMint
class node_affinity_guest toneRose
class node_user,node_vllm_engine toneIndigo
```

## Components

### Serving entrypoints

- [vLLM CLI hook](https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentcache/entrypoints/cli/main.py): `agentinfer/agentcache/entrypoints/cli/main.py`
- [Serve takeover](https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentcache/entrypoints/cli/serve_profile.py): `agentinfer/agentcache/entrypoints/cli/serve_profile.py`
- [Cache controller](https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentcache/core/controller.py): `agentinfer/agentcache/core/controller.py`
- [Cache scheduler](https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentcache/core/scheduler.py): `agentinfer/agentcache/core/scheduler.py`
- [vLLM adapter](https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentcache/core/api_adapter.py): `agentinfer/agentcache/core/api_adapter.py`

### Scheduling runtime

- [Request identity](https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/scheduling/identity.py): `agentinfer/scheduling/identity.py`
- [Program scheduler](https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/scheduling/runtime.py): `agentinfer/scheduling/runtime.py`
- [Progress TTL policy](https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/scheduling/progress_ttl/strategy.py): `agentinfer/scheduling/progress_ttl/strategy.py`
- [Scheduling contracts](https://github.com/openjiuwen-ai/agent-infer/tree/main/agentinfer/scheduling): `agentinfer/scheduling`
- [Backend facts](https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/scheduling/backend.py): `agentinfer/scheduling/backend.py`

### AgentBench

- [Benchmark runner](https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentbench/benchkit/runner.py): `agentinfer/agentbench/benchkit/runner.py`
- [Agent runners](https://github.com/openjiuwen-ai/agent-infer/tree/main/agentinfer/agentbench/agents): `agentinfer/agentbench/agents`
- [Trace replay](https://github.com/openjiuwen-ai/agent-infer/tree/main/agentinfer/agentbench/replay): `agentinfer/agentbench/replay`
- [Request proxy](https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentbench/request_proxy/server.py): `agentinfer/agentbench/request_proxy/server.py`
- [Request traces](https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentbench/request_proxy/request_trace.py): `agentinfer/agentbench/request_proxy/request_trace.py`
- [Session analysis](https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentbench/benchkit/session_analysis.py): `agentinfer/agentbench/benchkit/session_analysis.py`
- [Metrics collection](https://github.com/openjiuwen-ai/agent-infer/tree/main/agentinfer/agentbench/benchkit/metrics): `agentinfer/agentbench/benchkit/metrics`

### Router affinity

- [Affinity guest](https://github.com/openjiuwen-ai/agent-infer/blob/main/agentinfer/agentrouter/agent_hint_affinity/src/lib.rs): `agentinfer/agentrouter/agent_hint_affinity/src/lib.rs`

### Other components

- **Serving operator**
- **vLLM engine**
- **vLLM Router**

## Connections

- Serving operator → vLLM CLI hook: invokes (evidence: `README.md`)
- vLLM CLI hook → Serve takeover: routes serve (evidence: `agentinfer/agentcache/entrypoints/cli/main.py`)
- Serve takeover → vLLM engine: activates serving (evidence: `README.md`)
- Cache controller → Program scheduler: builds scheduler (evidence: `agentinfer/agentcache/core/controller.py`)
- Cache controller → Progress TTL policy: builds policy (evidence: `agentinfer/agentcache/core/controller.py`)
- Cache scheduler → Program scheduler: adapts scheduling
- vLLM adapter → Program scheduler: adapts requests
- Program scheduler → Request identity: uses identity (evidence: `agentinfer/scheduling/runtime.py`)
- Program scheduler → Progress TTL policy: dispatches decisions (evidence: `agentinfer/scheduling/runtime.py`)
- Program scheduler → Scheduling contracts: uses contracts (evidence: `agentinfer/scheduling/runtime.py`)
- Program scheduler → Backend facts: reads backend facts (evidence: `agentinfer/scheduling/runtime.py`)
- Benchmark runner → Agent runners: runs agents (evidence: `README.md`)
- Benchmark runner → Trace replay: replays traces (evidence: `README.md`)
- Request proxy → Request traces: records requests (evidence: `agentinfer/agentbench/request_proxy/server.py`)
- Session analysis → Request traces: reads facts (evidence: `agentinfer/agentbench/benchkit/session_analysis.py`)
- Metrics collection → Benchmark runner: collects metrics
- Affinity guest → vLLM Router: adds affinity (evidence: `README.md`)
