Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM
MarkTechPost
Buoyant has released Kinference, an open-source inference control plane for Kubernetes and vLLM.
Buoyant has launched Kinference, an Apache 2.0 open-source inference control plane designed for vLLM running on Kubernetes. The tool introduces features such as declarative routing policies, KV-cache-aware load balancing, and per-tenant metrics. According to Buoyant's internal benchmarks, the control plane successfully reduced median Time-to-First-Token (TTFT) by 95%.
Why it matters: The release of Kinference provides Kubernetes and vLLM users with advanced routing and load balancing capabilities that can drastically reduce model response times.
