One robots.txt Line Now Opts You Out of Four Big AI Crawlers

Short answer: On October 30, 2026, OpenAI, Google, Microsoft and Anthropic said they will honor a single shared robots.txt signal called __usage_ai-training, letting a site opt out of AI training across all four providers with one rule instead of one rule per crawler. None of them honors it yet, and it only covers training, not crawling.

In this post: 5 things — what the token is, what it does not do, who ships it when, the fine print that can backfire, and what publishers are already doing

Four of the biggest AI companies just agreed to read the same line of text. That sounds small until you remember what publishers have been doing since 2023: naming GPTBot, then Google-Extended, then ClaudeBot, then Applebot-Extended, then CCBot, then Meta-ExternalAgent, one robots.txt group at a time, forever.

On Friday, OpenAI, Google and Microsoft each published a version of the same announcement. Anthropic backed it too, and one of its own standards people wrote the spec. One token, four crawlers, one switch.

Here's the line, straight from page 2 of the draft:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
3 lines. No new syntax.

That reads as "no AI training anywhere except the press kit." Under the proposal, a participating company checks that group before it decides what your page may be used for. It's the same path matching robots.txt has used for 30 years, so nothing about the grammar changes.

What the token does, and the 3 things it doesn't

Honestly, this is where most coverage has been sloppy, so let me be blunt. The token is a statement of preference about use. It is not a lock on the door.

The draft says it plainly: "This policy token is not a directive for crawling behavior." A bot that honors it may still fetch your page. If you want it to stop fetching, you keep using that crawler's own name in robots.txt, exactly as you do today.

Three more limits sit inside the document. Wildcard groups don't count, so a blanket User-Agent: * block expresses nothing about training. A file with no token group expresses nothing at all. And private licensing deals can override the preference, which is the escape hatch all 4 companies wanted and got.

Then the one that will surprise people: nothing is retroactive. Your pages are already in somebody's training set, and a robots.txt edit on October 31 does not pull them out of a model that shipped in September.

OpenAI frames the whole thing as "a coordination mechanism rather than a security control," and that's the honest description. It works because four companies say they'll read it. That's it. Scrapers outside those four can keep ignoring your file, and some already do.

Who switches it on, and when: 5 timelines

This is the part that decides whether the announcement means anything this year. The timelines are wildly uneven, and only one company gave an exact date.

Company	Status	When	Control that keeps working
Anthropic	Building it now	Started	ClaudeBot
Microsoft	Dated commitment	February 1, 2027	Bing training and grounding tags
OpenAI	Commitment, no date	"Coming months"	GPTBot
Google	Commitment, no date	Early next year	Google-Extended
Meta	Support only	None given	Nothing new

Microsoft went furthest and it's not close. Its Bing post adds page-level training tags, a separate page-level "AI Grounding" setting, and a third option called DisallowExtended that allows your content in Copilot, AI Mode and Microsoft 365 but keeps it out of the grounding services Microsoft sells to other companies.

The transition has teeth. Old NoArchive and NoCache tags stop being read as preferences after March 1, and temporary grounding controls open in Bing Webmaster Tools on December 15 for sites still using them. Miss the window and pages "default to Allow for grounding until Microsoft recrawls them."

Google's version is narrower but has one detail worth knowing. If you opt out with the token and don't mention Google-Extended anywhere, Google says it will treat that as if you had disallowed Google-Extended, the control it has offered since 2023. Existing per-vendor rules keep running alongside the new signal, so you can make a broad choice and then carve out one provider.

Meta's communications head Andy Stone called a common signal "an important step" and stopped there. No date, no commitment. Which is a strange look, given that Meta's own revised commercial terms, pushed back to November 20 after publisher backlash, would let it crawl sites tied to business accounts.

2 pieces of fine print that can backfire

An independent read of the draft, run against three robots.txt parsers and 61 live files, found a trap worth repeating. A token group that blocks only one folder doesn't mean "block this folder." It means "block this folder and approve training on every other path," because RFC 9309 treats unmatched paths as allowed, and the draft reads an allowed path as a preference in favor.

So this 2-line version is wrong if your intent is partial:

User-agent: __usage_ai-training
Disallow: /private/
1 folder blocked, everything else silently approved.

Write the whole intent instead: block the root, then list the exceptions. Same analysis found zero of those major sites carrying the token, which is fair, since the draft was hours old. Forty-six of those 61 already name at least one training crawler by hand, and nytimes.com names all 24 tokens on the list.

There's a second wrinkle. The IETF's own working group has a competing draft that puts the preference in a Content-Usage rule inside a crawler's group, and the two designs disagree on 4 points, including whether a wildcard group counts. The token version won the announcement. It hasn't won the standards fight.

Nate Hake, who put in more than 100 hours in those meetings, was quick to say so publicly: "it's not actually done at the ietf level," and he's unhappy that the current draft exempts anything a user does, which he says hurts creators getting hit by AI copycats.

Publishers already voted, and they voted by hand

The reason this announcement landed is that manual blocking got out of control. A BuzzStream audit of the 100 largest US and UK news sites this year found 79% disallow at least one training bot, 14% block all 11 it checked, and 18% block none. Seventy-one percent also block at least one retrieval crawler, the kind that cites them.

The gap between publishers and everyone else is enormous. HasData measured 10,894 domains in July: 56.4% of news publishers block at least one AI crawler in robots.txt, against 10.3% of the top web. For GPTBot alone it's 50.5% versus 7.9%.

And a lot of those blocks are decorative. Of 592 sites that disallowed GPTBot, 234, or 39.5%, still handed the crawler a live page when it knocked. Cloudflare Radar's September snapshot of 4,242 files shows 8.11% fully disallowing GPTBot, up from 7.15% a year earlier, while explicit Allow rules for GPTBot grew from 2.51% to 7.69%. Blocking isn't even the trend anymore. Selective allowing is.

Which explains why the labs are doing this now. Anthropic's crawl-to-referral ratio fell from 36,551 pages per referral a year ago to 988 in the third quarter, and 13.47% of AI-bot requests now come back 403, more than double the 5.98% of a year earlier. Edge blocking is eating them.

Meanwhile the legal fight is live. The publisher suits against OpenAI and Microsoft are at summary judgment before Judge Sidney Stein, the Justice Department filed a statement of interest backing OpenAI on September 1, and the News/Media Alliance filed an amicus brief for the publishers that same morning. Its CEO, Danielle Coffey, calls the case "existential for the news media industry."

Meta is the counter-example in plain sight. Its revised commercial terms would let it crawl sites tied to business accounts, the News/Media Alliance called them "outrageous and probably illegal," and the UK's competition regulator opened a formal probe into the terms the same day. Support for a voluntary token while pushing that through is a position, not a contradiction anyone has to explain.

What this means for you: 4 moves

If you run a site and you care about this, do four things. Keep your existing per-vendor groups, because nothing new is being honored yet. If you add the token, write it as a root block plus exceptions. Don't assume it stops a crawl. And watch February 1, because that's the first date anyone can actually test whether a big provider reads the line.

If you're a reader rather than a site owner, the practical effect is close to zero today. Your favourite news site can already block training crawlers by name, and 79% of the biggest ones do. What changes is the maintenance cost, not the outcome.

My read: I think the useful part isn't the token, it's the definition. Four companies just agreed that "AI training" means "using an asset to modify the learned parameters of a generative AI model," and that ranking or classification models don't count. That definition travels into the EU AI Act's code of practice, which names robots.txt as the primary mechanism under Article 53(1)(c). A shared vocabulary is worth more long term than a shared switch.

What it won't fix is the thing publishers are angriest about. Training is the old wound. AI answers that summarise your article without sending a click are the current one, and this token says nothing about them. Microsoft is the only one of the 4 shipping a separate grounding switch, and it's honest about the trade: "Neither choice guarantees inclusion, traffic, or payment."

People Also Ask

Is the shared AI training opt-out live right now?
No. Anthropic says it has started implementing support, OpenAI targets "the coming months," Microsoft honors it from February 1, and Google says early next year. Adding the token today costs you nothing and changes nothing yet.

Does blocking AI training remove my site from ChatGPT or Gemini answers?
Not by itself. Training and grounding are separate settings. Google says sites that allow Google-Extended stay eligible to appear in Gemini app answers, and Microsoft keeps Bing search eligibility unchanged regardless of your preference.

Will a User-Agent: * block count as an opt-out?
Under this draft, no. RFC 9309 wildcard groups must not be read as matching the policy token, which was deliberate, so a site blocking everything with a wildcard has still said nothing about training until it adds the token group.

Does this help with content already scraped?
No. The preference applies to the robots.txt in force when a page was fetched, so it governs future collection. Anything already in a dataset or baked into model weights stays there.

By Dibakar Munshi, news-ofai.blogspot.com

📚 Sources

#	Source	Used for	Link
1	OpenAI Blog	Announcement, "coming months" timing, existing controls	openai.com
2	IETF Datatracker	Draft text, token rules, wildcard exclusion, expiry	datatracker.ietf.org
3	Microsoft Bing Blog	February date, grounding and DisallowExtended, March deadline	blogs.bing.com
4	Google Search Central	Google-Extended fallback, timing, Gemini grounding	developers.google.com
5	The Verge	Meta's statement, four-company scope, RSL and Cloudflare context	theverge.com
6	HasData AI Crawler Block Index	56.4% vs 10.3% blocking, 39.5% paper-only bans	hasdata.com
7	TechnologyChecker (Cloudflare Radar data)	8.11% GPTBot disallow, crawl-to-refer ratios, 403 share	technologychecker.io
8	Media Factual / BuzzStream audit	79% of top 100 news sites block a training bot	mediafactual.com
9	SXZ.io draft analysis	61-file survey, partial-group trap, parser tests	sxz.io
10	News/Media Alliance	Amicus brief, Meta terms statement, Coffey quote	newsmediaalliance.org