Together AI has launched Together Compiler, a system that uses teams of AI agents to rewrite the GPU code behind the AI models it serves — tuning new open-source models overnight instead of over weeks, with no work required from customers. Why it matters: inference is where AI companies make or lose their margins, and Together says its agent-written code now beats kernels tuned by human experts — a concrete, production-scale step toward AI systems that improve the systems that run AI.
November 4, 2026 — San Francisco
What Together AI announced
Together Compiler is the first project in a larger initiative Together AI calls Polymorphic Inference — its term for inference that keeps adapting on its own as models, workloads, and chips change. The launch is not a research demo: the company says the compiler is already running across many of its most popular serverless endpoints, which serve trillions of tokens per day. Customers on dedicated deployments can join a limited beta, with self-serve access coming soon. A research paper accompanies the launch.
The bottleneck the system attacks is validation. In traditional setups, Together says, a single optimization an agent proposes can take up to 15 minutes to check inside an inference engine, which starves the GPUs doing the checking. The compiler’s design is built to maximize how fast agents can iterate — the company says it runs 252 times more experiments than existing systems that use agents for inference optimization.
The results Together is claiming
All of the following figures are Together AI’s own reported results, not independently verified. On Kimi K3 and GLM 5.3, the company says the compiler reached 1.5x and 1.4x higher peak throughput, respectively, than the best result on SemiAnalysis’s InferenceX, an open inference benchmark. On a 1-billion-parameter Llama model, it says agent-written optimizations passed 2,100 tokens per second and ran up to 1.7x faster than hand-written megakernels — including Together’s own — at every output length tested, on both NVIDIA H100 and B200 GPUs. MiniMax M3.1 Flash reportedly hit 1,111 tokens per second on B200 hardware (1.5x the baseline), and DeepSeek v4.1 Flash 1,205 tokens per second on GB300 (1.2x the baseline).
One customer is named: Phonic co-founder Moin Nadeem says running the compiler cut his product’s median time to first speech by nearly 50%, overnight. The context Together cites is model churn — it says 80% of OpenRouter traffic now goes to models released in the last three months, and each of the ten most popular open models there uses a different architecture, which is why hand-tuning keeps falling behind.
Why it matters
Every new open model currently arrives with the same tax: weeks of expert engineering before it runs at full speed, by which time the next model is already out. If agents can reliably close most of that gap in a day, the economics of inference shift toward whoever automates the tuning — and the advantage of hand-crafted kernels, a real moat for some providers, shrinks. The honest caveats: these are the vendor’s numbers, the speedup varies widely by model (1.2x to 1.7x in the detailed results), and “recursive self-improvement” here means agents optimizing inference code, not models redesigning themselves. Even with those caveats, a self-tuning inference stack running at trillion-token scale is a real milestone, and rivals serving open models will have to answer it.
Frequently asked questions
What is Together Compiler?
An optimization system from Together AI that uses teams of AI agents to rewrite the GPU code behind hosted AI models, continually tuning inference performance as models and workloads change.
How much faster does Together Compiler make AI models?
In Together AI’s own testing, up to 1.7x faster than hand-written kernels on a small Llama model, and 1.4x–1.5x higher peak throughput than the best published InferenceX results on Kimi K3 and GLM 5.3. The company headlines gains of up to 2.6x over hand-written expert code across its tests.
Can customers use Together Compiler now?
It is already live across many of Together AI’s popular serverless endpoints. Dedicated-deployment customers can join a limited beta, and the company says self-serve access is coming soon.
Sources: Together AI announcement via BusinessWire (November 4, 2026).