Four AI Labs Back One Robots.txt Token to End AI Training Opt-Out Fragmentation

OpenAI, Google, Microsoft, and Anthropic have jointly committed to honoring a single __usage_ai-training signal in robots.txt — a move that, when fully implemented, will let any web publisher opt out of AI training-data collection across all four organizations with one entry instead of four. The announcement, made October 30, 2026, centers on an individual Internet-Draft filed by Anthropic engineer Kevin Kelley through the IETF's AI Preferences working group, and marks the first time the four dominant frontier-AI developers have aligned on a shared machine-readable preference for training data. The commitment does not retroactively affect data already collected.

For publishers, the practical significance of this shift can be measured in the clutter it eliminates. Today, preventing a site's content from reaching AI training pipelines requires separate robots.txt entries for GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), Bingbot variants and the NoArchive/NoCache tags (Microsoft), along with a growing list of third-party scrapers operating under each lab's infrastructure umbrella. A Cloudflare analysis of nearly 4,300 sampled robots.txt files from Q1 2026 found that roughly 13.8 percent already block GPTBot, 11.5 percent block ClaudeBot, and 10.7 percent block Google-Extended — each through separate, manually maintained entries. A Reuters Institute study found that 48 percent of widely-used news sites across ten countries block OpenAI's crawlers, and a separate arXiv study tracked AI-blocking by reputable publishers rising from 23 percent in September 2023 to nearly 60 percent by May 2025. The fragmentation is real, and it is growing.

How the __usage_ai-training Token Works in Practice

The new signal uses the existing robots.txt User-agent: grouping mechanism, adding no new syntax to the protocol. A site operator placing the following two lines opts the entire domain out of AI training across all four participating crawlers:

User-agent: __usage_ai-training
Disallow: /

Operators can carve out exceptions using the standard Allow: directive — permitting a press kit or investor relations section while blocking the rest, for example. The token targets training use only; separate directives control AI retrieval and search grounding, meaning blocking training does not by itself remove a site from AI-powered search features. The draft document, draft-kelley-aipref-training-preference-signal-00, was authored by Kelley and filed October 30, 2026, with an expiration date of May 3, 2027 pending renewal.

Critically, the signal is a preference statement, not a technical block. The robots.txt standard has always operated this way — it is a protocol of declared intent, not an enforcement mechanism. The draft text says crawlers "MAY" respect the preference, not "MUST." A TollBit report on AI crawler behavior in Europe in 2025 found approximately 15 percent of identified AI page fetchers still reached URLs marked as disallowed. Compliance is voluntary and historically imperfect.

A Broader IETF Standards Process, With This Draft as One Piece

The announcement has been widely described as IETF-backed, which is accurate in the sense that it is filed through the IETF process, but needs clarification. The IETF's AI Preferences working group — chartered in January 2025 and currently developing two adopted drafts, one covering a vocabulary for expressing AI usage preferences and another covering attachment mechanisms — is the institutional context for this work. RFC 9969, published in May 2026, is the report from the IAB's September 2024 AI-CONTROL Workshop that gave conceptual weight to this effort; a workshop participant's concluding remark that "robots.txt is broken and we broke it" captured the industry's mood precisely.

Kelley's October 30 draft is a separate individual submission — not yet adopted by the working group and not a ratified IETF standard. It has no formal IETF standing until and unless the AIPREF working group takes it up and the IESG approves it. The companies' commitment to honor it is voluntary and pre-dates any formal standardization. What exists today is a coordinated pledge by four major AI developers, anchored in a living document that expires in six months absent renewal. That is a meaningful step forward from the current fragmented landscape, but it is not a finalized internet standard.

Staggered Rollout: What Each Company Has Actually Committed

The four commitments are real but not simultaneous. Anthropic has begun work on honoring the token, though support will complement rather than replace its existing ClaudeBot controls, and a precise date has not been given. OpenAI has said it will add support "in the coming months." Microsoft has published the most specific timeline: it will honor the preferences starting February 1, 2027, and has said its implementation will replace the NoArchive and NoCache tags it has relied on since 2023 for similar purposes. Google plans to support the token in early 2027 and has noted in its crawler documentation that the signal aligns with the EU's Code of Practice for general-purpose AI under the AI Act, which formally recognizes robots.txt as a valid machine-readable opt-out mechanism for rights holders.

Until each company activates support, publishers relying solely on __usage_ai-training will not be protected against that company's crawlers. The existing per-vendor directives — GPTBot, ClaudeBot, Google-Extended, OAI-SearchBot — remain the operative controls through the transition period. Publishers who want protection during the rollout should maintain their current per-vendor rules in parallel.

Hidden Tradeoffs: Grounding, Visibility, and Microsoft's Split Controls

The opt-out surface is not as clean as the single-token framing implies, and two platform-specific complications deserve attention.

Google has indicated that a site using __usage_ai-training without an explicit Google-Extended allow directive may be treated as opting out of Google-Extended, which governs not only training but also grounding — the process by which Gemini surfaces citations and sends clicks back to original publishers. Publishers who want to block Google from training on their content while remaining eligible for Gemini citation will need to maintain a nuanced robots.txt configuration: a __usage_ai-training Disallow alongside an explicit Google-Extended Allow.

Microsoft's situation differs in structure. Alongside the shared training token, Microsoft is introducing a separate grounding control called "AI Grounding," which governs whether Microsoft's AI products fetch and cite a page in response to a user query. Notably, this grounding control offers a split option: a publisher can block third-party AI tools built on Microsoft's infrastructure while still allowing Microsoft's own Copilot to cite the page. This level of granularity does not currently exist in Google's implementation. The trade-off is that Microsoft's defaults favor AI access — both the training and grounding preferences are set to Allow unless explicitly changed.

The Coverage Gaps That Remain

Four companies is not the whole market. Meta has publicly expressed support for a common opt-out signal — Meta's communications head called it "an important step" — but has not committed to implementing the __usage_ai-training token or any specific timeline. Amazon and Apple have not been named in any related discussions. Google's AI Overviews, which draw on separate infrastructure from the Google-Extended crawler, sit outside the scope of this token. Private licensing agreements between publishers and AI companies can override the stated preference entirely. And the entire commitment leaves untouched the legal question of whether training on copyrighted content without opt-out protections already required a license — a matter still working through courts in multiple jurisdictions.

What the announcement advances is the expression layer of the opt-out problem: getting a preference communicated to multiple crawlers with minimal publisher effort. What it leaves to future work is everything harder — auditability, enforcement, retroactive redress, compensation, and scope expansion to the many AI developers who are not among the original four signatories.

The next meaningful milestones to watch are Microsoft's February 1, 2027 activation and early 2027 Google enforcement, which will provide the first real-world test of whether the coordinated pledge translates into consistent crawler behavior. If the AIPREF working group formally adopts a version of Kelley's draft and the IESG advances it to Proposed Standard, that will carry substantially more weight than the current voluntary commitment. Until then, publishers face a transitional window during which the signal exists, four labs have promised to honor it, and the practical protection it provides depends entirely on each lab's execution of its own rollout schedule.