The llm-d project released version 1.0 on October 20, 2026 — the first production-ready version of the open-source, Kubernetes-native stack for running large AI models on your own infrastructure, backed by IBM, Red Hat, Google, CoreWeave and NVIDIA. Why it matters: as AI agents multiply the number of model requests a company serves, inference — not training — is becoming the bill that matters, and llm-d is the open camp’s bid to keep that bill on hardware you control.
October 20, 2026
What happened?
Contributors announced llm-d 1.0 at the PyTorch Conference North America in San Jose. The project was co-founded last year by IBM, its Red Hat subsidiary and Google — the lead maintainers — together with CoreWeave and NVIDIA, and now counts more than 500 contributors across over 25 organizations, including AMD, Cisco, Hugging Face, Intel and Mistral AI. It is hosted as a sandbox project by the Cloud Native Computing Foundation and works alongside serving engines such as vLLM and SGLang rather than replacing them.
What is new in 1.0?
The individual speed tricks are not new — routing each request to the replica that already holds its context in cache, tiering that cache across GPU memory, CPU memory and storage, splitting prompt processing from answer generation, and spreading large mixture-of-experts models across accelerators. What 1.0 adds is the operational layer enterprises asked for: a stable API, quality-of-service controls so multiple tenants with competing priorities can share the same infrastructure, flow controls that keep bursts of agent activity from starving interactive users, and upgrades that do not interrupt service. In Google’s conference demonstration, 1,000 sandboxed agents hit a single llm-d endpoint across vLLM, SGLang and a mixed fleet of TPUs and GPUs.
What do the backers claim it saves?
The savings figures are the backers’ own, not independent benchmarks. IBM says its router captured 97% of available cache reuse in a recent agentic coding test, and that IBM and Red Hat ran 3,000 software-engineering agents concurrently on self-hosted infrastructure at a cost five to ten times lower than serving the same open model through a hosted service. IBM Research also cited an 11.8x efficiency advantage in a quantum error-correction hardware design problem. Treat those as vendor-reported results until third parties reproduce them.
Why it matters
Agent workloads break the old serving model: one task can mean dozens of model calls that share long stretches of context, arrive in bursts, and stall if the system recomputes everything from scratch. llm-d’s pitch is that a coordinated pool of model servers beats a pile of isolated endpoints — and that companies should be able to get that coordination without locking themselves to one model provider, one cloud or one chip vendor. If 1.0 holds up in production, “run it ourselves” becomes a realistic default for enterprise AI again, not a science project.
Frequently asked questions
What is llm-d?
An open-source, Kubernetes-native framework for distributed LLM inference — it coordinates how large models are served across a cluster of machines.
Who is behind it?
IBM, Red Hat and Google are the lead maintainers; CoreWeave and NVIDIA co-founded the project, which now has more than 500 contributors from over 25 organizations and is hosted by the Cloud Native Computing Foundation.
Does it replace vLLM or SGLang?
No. It sits around those serving engines, adding distributed routing, cache management and operational controls.
What is actually new in version 1.0?
Production readiness: a stable API, multi-tenant quality-of-service controls, priority flow control for agent bursts, and zero-downtime upgrades.
Sources: SiliconANGLE, Unite.AI, Google Cloud Blog (all October 20, 2026)