Rohan Paul

Agent traffic is a strange workload for an inference server. Each turn resends a long, nearly identical prefix (system prompt, tool schemas, history) and asks for a short tool call back. At DigitalOcean’s Open Intelligence Summit, Yifan Qiao of Inferact is giving a Serve track session on running vLLM on agentic production workloads. That shape of traffic is why prefix caching and KV-cache reuse matter more for agents than raw tokens per second. A cache miss on a long prefix means paying for the full prefill again, on every turn of a long task.