One robots.txt Line to Opt Out of AI Training: OpenAI, Google, Microsoft and Anthropic Back __usage_ai-training
On October 30, 2026, OpenAI, Google and Microsoft said they will honor a shared robots.txt token, __usage_ai-training, drafted at the IETF by Anthropic's Kevin Kelley. One group in robots.txt can now tell every participating AI lab not to train on your site. Here is the exact syntax, each company's timeline, Bing's new grounding controls and what the token cannot do.
What changed on October 30, 2026
On October 30, 2026, OpenAI, Google and Microsoft each announced support for one shared signal in robots.txt that tells AI companies whether they may use a website's content to train generative AI models. The signal is a new policy token, __usage_ai-training, defined in an IETF Internet-Draft written by Kevin Kelley of Anthropic, building on the work of the IETF AI Preferences (AIPREF) working group. This page is the long-form briefing behind the BSH Technologies Instagram carousel.
- What it is: a single robots.txt token that expresses a yes or no on AI training to every participating provider at once.
- Who backs it: OpenAI, Google, Microsoft and Anthropic, whose engineer wrote the draft.
- Why it matters: until now, site owners had to block each crawler one by one (GPTBot, Google-Extended, ClaudeBot and others) and keep that list up to date.
It lands in a month packed with frontier releases, from OpenAI's GPT-6.1 Terra to Google's Gemini 4 Carbon, all of which depend on web data.
The exact robots.txt syntax
The draft adds no new directives and no new grammar. You write an ordinary robots.txt group, but the User-Agent line names the policy token instead of a crawler. Disallow means "do not use this path for AI training"; Allow means training is fine. This is Figure 1 from the draft:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
Key rules from the draft text:
- Wildcards do not count. A
User-Agent: *group MUST NOT be read as matching the token. If you want to opt out of training, you have to name__usage_ai-trainingexplicitly. - No line, no preference. If your robots.txt has no
__usage_ai-traininggroup, no preference has been expressed. - Normal path rules apply. Path matching and longest-match precedence work exactly as in RFC 9309, so you can opt whole sites or single folders in or out.
- It is about use, not crawling. The token is "not a directive for crawling behavior". To limit crawling, keep using each crawler's own product token.
- Specific beats general. Company-specific tokens such as Google-Extended may be treated as more specific, and private agreements may override the shared preference.
"AI training" follows the AIPREF definition: using content to modify the learned parameters of a generative AI model. Training models used only for classification, ranking or scoring is not covered.
Each company's timeline
OpenAI: "in the coming months"
OpenAI says it "will honor a common signal in robots.txt" and aims to implement it "in the coming months", with details to follow. Its existing crawler controls, such as the GPTBot rules in robots.txt, keep working alongside the new token.
Google: in 2027, read as a Google-Extended block
Google's Search Advocate John Mueller wrote that when a site uses the token to opt out of training and Google-Extended is not explicitly specified, Google "plans to interpret that as if the publisher had disallowed Google-Extended". Google aims to support the control "along with other AI providers in 2027". Sites that explicitly allow Google-Extended stay eligible to appear in Gemini Apps responses through grounding. Google also notes the approach is consistent with the EU Code of Practice for general-purpose AI models, which recognises robots.txt as a primary opt-out method.
Microsoft: from February 1, 2027, plus new grounding controls
Microsoft goes furthest. Starting February 1, 2027, Bing's crawlers will honor:
- AI Training in robots.txt (the shared token), plus a page-level version that can override the robots.txt setting for a single page.
- AI Grounding, a separate page-level preference that controls whether a page can inform generated answers in Bing, Copilot and Microsoft 365. A new
DisallowExtendedvalue limits grounding to Microsoft's own experiences and excludes third-party grounding services. - Search is untouched. Microsoft says these choices do not change eligibility for Bing search results.
The new preferences replace the old NoArchive and NoCache signals. Microsoft asks sites to act by March 1, 2027, including recording a temporary grounding preference in Bing Webmaster Tools; after that date NoArchive and NoCache are no longer read as training or grounding preferences.
Anthropic and Meta
The draft's author, Kevin Kelley, is a technical lead for standards at Anthropic. According to The Verge, Anthropic has already adopted the signal. Meta's communications head Andy Stone said Meta supports a common signal, but did not say whether Meta will honor it.
The catch: a request, not a lock
- It is voluntary. robots.txt has never been enforceable. Scrapers that ignore it can still copy your pages.
- It is not retroactive. The preferences cover future training; models already trained on your content stay as they are.
- It covers training only. It does not stop crawling, search indexing or, outside Bing's new controls, use of your pages to ground AI answers.
- It is still a draft. The Internet-Draft is valid until May 3, 2027 and may change as the AIPREF working group continues.
For context on how fast AI labs are shipping new models that need this data, see our roundup of 18 AI launches in one month, Mistral's open-weight Le Chonk and Microsoft's Agensh agent harness.
What to do with your robots.txt
- Decide per section. Opt the whole site out, or keep folders such as a press kit or docs open with
Allowlines. - Add the group explicitly. Remember that
User-Agent: *does not cover the token. - Keep your crawler rules. GPTBot, Google-Extended and other company tokens still work and can be more specific.
- If you use Bing: move off NoArchive and NoCache and set training and grounding preferences before March 1, 2027.
Sources
- OpenAI: Making it easier for websites to express AI training preferences
- Google Search Central Blog: Supporting a shared AI training control for websites
- Microsoft Bing Blog: Your content, your choice: clearer preferences for training and grounding
- IETF Internet-Draft: Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00)
- IETF AI Preferences (AIPREF) working group
- The Verge: The biggest names in AI are backing a new way to opt out of scraping
Frequently asked questions
What is __usage_ai-training?
It is a shared robots.txt policy token defined in the IETF Internet-Draft draft-kelley-aipref-training-preference-signal-00. A robots.txt group with User-Agent: __usage_ai-training and Disallow rules tells participating AI providers not to use those paths to train generative AI models.
How do I opt my whole site out of AI training?
Add a group to robots.txt with the line User-Agent: __usage_ai-training followed by Disallow: /. You can add Allow lines, such as Allow: /press-kit/, to keep specific folders available.
Does User-Agent: * cover the new token?
No. The draft says wildcard groups must not be interpreted as matching the policy token, so you have to name __usage_ai-training explicitly.
Which companies will honor it, and when?
Microsoft will honor it from February 1, 2027. Google plans to support it in 2027 and will read an opt-out as a Google-Extended disallow when Google-Extended is not specified. OpenAI says it will implement it in the coming months. Anthropic, whose Kevin Kelley wrote the draft, has already adopted it. Meta is supportive but has not committed.
Does it stop crawling or remove my content from existing models?
No. The token expresses a preference about AI training only. It is not a crawling directive, it is voluntary, and it does not change models that were already trained.
From the blog
View all posts
Mistral Large 4 "Le Chonk" Goes Open Weight: 1T Parameters, Top Open Coder, Built for Cyber Defense
On October 29, 2026, Mistral released the weights of Mistral Large 4 (Le Chonk), its 1-trillion-parameter flagship. It leads open models on FrontierCode 2.0 (19.7%) and Terminal-Bench 4.0 (41%), matches GLM-5.3 on the AA Coding Agent Index with 47% fewer tokens, and refuses 96.6% of malicious cyber requests. Here are the specs, scores, licence, hardware and the catch.
Google's Gemini 4 Carbon: #1 Frontier Model, Built for Cyber Defense, and the Catch
On October 28, 2026, Google announced Gemini 4 Carbon, its strongest model yet. It ranks #1 on the Artificial Analysis Intelligence Index v5.0 at 59, leads SEC-bench Pro (88.2%), ExploitGym (54.8%) and CWE-bench (71.0%), and blocked every attack in Gray Swan's prompt-injection test. For now it goes only to vetted cyber defenders. Here are the benchmarks, the safety numbers, the price and the catch.