OpenAI, Google, Microsoft and Anthropic announced support on October 30, 2026, for a shared robots.txt signal that would let website operators communicate AI-training preferences across participating providers. Their commitments come with different implementation schedules, rather than an immediate, universal rollout. (openai.com)
The proposed token, code>__usage_ai-training/code>, concerns how content is used for training, not a blanket instruction to stop crawling. It remains an individual Internet-Draft, not an approved Internet Engineering Task Force standard. Its practical effect depends on providers implementing and honoring the preference; publishing it does not itself prevent scraping. (datatracker.ietf.org)
Different commitments and rollout dates
Google says it aims to support the control in 2027, without specifying a month in its announcement. OpenAI targets implementation in the coming months and says it will publish further implementation details. OpenAI describes the initiative as voluntary and says its existing robots.txt crawler controls will remain in place. (developers.google.com)
Anthropic has already started implementing support, according to an October 30 message from its employee Kevin Kelley, who authored the proposal. That is not confirmation of completed deployment. Kelley says adding support across the company’s services will take time and that the new mechanism complements existing crawler controls. (mailarchive.ietf.org)
Microsoft says its new training and grounding preferences will take effect on February 1, 2027, as it crawls. It is adding a separate preference for grounding—the use of web content to inform generated answers—alongside the shared training preference. Both default to allowing the use when no applicable preference is published. (blogs.bing.com)
Microsoft also says the preferences will replace its NoArchive and NoCache controls for training and grounding. Publishers relying on those older controls face a separate transition: Microsoft asks them to adopt the new preferences before March 1, 2027, to maintain their choices. That migration date is distinct from the February effective date. (blogs.bing.com)
A proposal, not an approved standard
Kelley’s proposal draws on work by the IETF’s AI Preferences working group, known as AIPREF. However, the IETF datatracker explicitly identifies it as an individual draft that is not endorsed by the organization and has no formal standing in its standards process. Company support should not be confused with standards approval. (datatracker.ietf.org)
The initial draft is dated October 30, 2026, and expires on May 3, 2027. That expiration is a document-lifecycle date, not a deadline for providers to deploy the signal or for the IETF to approve it. The proposal remains work in progress and may be revised or replaced. (datatracker.ietf.org)
For publishers, the intended benefit is simpler communication: one shared preference can address multiple participating providers rather than requiring a separate instruction for each. That does not mean every provider will interpret all related settings identically, or that existing company-specific controls are becoming unnecessary. Google and OpenAI explicitly say those controls will continue alongside the shared signal. (developers.google.com)
Training preferences are not a universal AI switch
Google’s treatment illustrates why provider-specific policies still matter. When a website opts out through the shared token and does not explicitly specify Google-Extended, Google plans to treat the choice as disallowing Google-Extended. Publishers can retain explicit instructions for Google while expressing a broader preference through the common signal. (developers.google.com)
Google also says websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks through grounding in Gemini Apps. Eligibility is not a traffic guarantee. Together with Microsoft’s separate grounding preference, that distinction shows why a training choice should not be described as controlling every use of content in AI-generated answers. (developers.google.com)
Compliance remains the limiting factor
The wider publishing industry has already identified the gap between expressing preferences and enforcing them. In its December 2025 digital-advertising playbook, the Interactive Advertising Bureau explains that robots.txt works only when bots choose to honor it. The organization discusses technical access controls and licensing arrangements as additional tools, rather than treating a published preference as sufficient protection. (iab.com)
The International Press Telecommunications Council’s March 2026 opt-out guidance likewise reflects a fragmented landscape. It incorporates Really Simple Licensing and Cloudflare’s Content Signals while advocating a formal, open standard developed through the IETF. That earlier assessment is broader context, not an endorsement of Kelley’s newly published proposal. (iptc.org)
The distinction for website operators is therefore between a common way to communicate a choice and a mechanism that guarantees compliance. The four providers’ commitments could reduce administrative work, but the immediate picture remains a staged rollout, continuing company-specific controls and a proposal still undergoing standards discussion. (openai.com)
