Buoyant launches Kinference: an open-source control plane for kubernetes and vllm inference

Buoyant, the developer of Linkerd, has introduced Kinference, an open-source inference control plane designed to manage platforms built on Kubernetes and vLLM. This system functions as a layer above the model server, applying routing policies, token rate limits, and KV-cache-aware load balancing across a fleet of vLLM workers. Kinference is licensed under Apache 2.0 and is currently available as version v0.1.0-alpha.7. Kinference operates by separating the data plane (vLLM) from the control plane (Kinference). It utilizes five Custom Resource Definitions (CRDs)—GatewayBackend, InferencePolicy, InferencePool, InferenceProvider, and InferenceWorkerSet—and monitors vLLM KV-cache events using NVIDIA’s dynamo-kv-router crate. Inference policies allow declarative rules to route or reject requests based on criteria like tenant or GPU pool state. Benchmarks conducted by Buoyant demonstrated significant performance gains over standard Kubernetes ClusterIP Service load balancing. On shared-prefix workloads, median end-to-end latency dropped by 45%, and peak goodput reached 103.0 requests per minute compared to 14.0 req/min for the baseline. The company notes that while competitors like llm-d and Dynamo excel in raw serving performance, Kinference's primary advantage lies in its governance features, including tenant accounting and provider fallback.