Google, Microsoft, OpenAI and Anthropic all announced on Friday, October 30, 2026, that they will voluntarily adopt a shared AI training control in robots.txt, so a website can state its training preferences once instead of separately for every AI company. Why it matters: it is the first time the biggest model builders have lined up behind one opt-out signal — but it is voluntary, still a draft, and Google does not plan to support it until 2027.
October 30, 2026 — The AI industry just agreed, in principle, on a single switch for publishers.
What was announced?
Each company announced it will adopt a proposal under consideration at the Internet Engineering Task Force (IETF), the global internet standard-setting body, built on work from its AIPREF working group. As Google’s John Mueller put it, instead of managing separate instructions for every AI developer, website owners will be able to use a shared “policy token” within robots.txt to set broad preferences across participating AI providers. Existing company-specific controls — such as Google-Extended, Google’s training control since 2023 — will continue to work alongside the new joint signal.
How will the shared control work?
If a website uses the policy token to opt out of training and does not explicitly mention Google-Extended, Google says it plans to interpret that as the publisher disallowing Google-Extended, and other providers can handle it the same way. Microsoft is going further on its own platform: it is separating training from “grounding” — use of content to generate AI answers — with independent preferences that can also be set per page. Microsoft says temporary grounding controls arrive in Bing Webmaster Tools on December 15, 2026, for eligible sites, that it begins honoring the new preferences on February 1, 2027, and that publishers using older directives have until March 1, 2027 to configure the new preferences. Microsoft’s message to publishers: you should not have to choose between being discoverable in Bing and deciding how generative models use your work.
When does it take effect?
Not immediately. Google said it will support the new control, along with other AI providers, in 2027, with updates on progress to come. The proposal is a draft, and Google notes it is consistent with the EU Code of Practice for General Purpose AI providers under the EU AI Act, which recognizes robots.txt as a primary way for publishers to state machine-readable preferences.
Why it matters
If you run a website, today’s reality is a patchwork: a different opt-out name for every AI crawler, with no guarantee any of them are honored. One shared token fixes the patchwork part — for the four biggest names, at least. The honest caveats are adoption and enforcement: this is voluntary, unfinished, and a year from switching on at Google. Everything AI, tested: we will treat this as a real standard only when the crawlers demonstrably obey it, and we will update our own robots.txt the day the token is final rather than on the strength of a press announcement.
Frequently asked questions
What is the shared AI training control?
A single “policy token” in a site’s robots.txt file that expresses AI training preferences across all participating providers at once, based on the IETF AIPREF working group’s proposal.
Which companies are adopting it?
Google, Microsoft, OpenAI and Anthropic each announced voluntary adoption on October 30, 2026.
Does it replace Google-Extended and other existing controls?
No. Company-specific controls continue to function alongside the shared signal. If a site opts out via the shared token without mentioning Google-Extended, Google plans to treat that as disallowing Google-Extended too.
When can publishers rely on it?
Google targets support in 2027. Microsoft begins honoring its new preferences on February 1, 2027, with a March 1, 2027 deadline for publishers migrating from older directives.
Sources: Search Engine Land, MLex