robots.txt AI Training Opt-Out Backed by OpenAI and Google

What is __usage_ai-training? It is a new policy token you place in the User-Agent line of a normal robots.txt group. Add Disallow: / under it and participating AI companies read that as "do not use this site to train generative models." It saves you from blocking GPTBot, ClaudeBot and Google-Extended one by one, but it is voluntary, and it does not cover AI answers.

What the four companies actually agreed to

On Friday, October 30, four of the largest AI companies said they will read one line in a website's robots.txt file. Put __usage_ai-training in a User-Agent group, add Disallow: /, and OpenAI, Google, Microsoft and Anthropic all say they will treat that as "don't train on this site."

One line instead of four. That's the whole promise, and it does fix a genuine chore. Right now a publisher has to keep a list of crawler names, from GPTBot to ClaudeBot to Google-Extended, and hope nobody quietly renames one.

The mechanism is written up in an Internet-Draft called "Machine-Readable Training Preference Signal," numbered draft-kelley-aipref-training-preference-signal-00. Anthropic's Kevin Kelley, a standards lead there, published it the same day. It adds no new directives. No new grammar either. It reuses the robots.txt path matching from RFC 9309, so a crawler that has never heard of the token just ignores the group, the way it ignores any unknown name.

Two words on the paperwork, because they matter later. The IETF's own page says this document "is not endorsed by the IETF and has no formal standing in the IETF standards process." It expires on May 3 unless someone revises it. The AI Preferences working group behind it has argued for more than a year, with chairs from Cloudflare and Cisco and participants from Meta, News Corp, the Financial Times, the BBC, the EFF and Creative Commons. A draft that four giants adopt on day one is a different animal from a standard, and the four giants know it.

Google's John Mueller spelled out how his company will read it: if you opt out with the token and don't mention Google-Extended anywhere, Google plans to treat that as if you had disallowed Google-Extended, the control it has offered since 2023. Same practical result, one less rule to maintain.

When each company starts: 4 timelines

This is where the announcement gets less exciting. Four companies, four different start dates, and only one of them is a date you could put in a contract.

Microsoft went furthest, and its blog post is the most honest document of the four. It splits training from grounding, grounding being the step where a model reads your page to answer a question right now, and gives you three settings per page: Allow, Disallow, and a new DisallowExtended that permits use inside Copilot, Bing AI Mode and Microsoft 365 but not in the grounding services Microsoft sells to other companies. It also warned that either restrictive option "will significantly reduce your content's visibility in AI-generated experiences," and that "neither choice guarantees inclusion, traffic, or payment."

There is a migration trap hidden in that post. Sites relying on the old NoArchive and NoCache tags have to publish the new preferences and, for grounding, record a temporary setting in Bing Webmaster Tools before March 1 next year. Miss it and your restriction quietly becomes an Allow.

OpenAI's announcement runs five short paragraphs and never once prints the token, though the word "signal" is a link to the draft. It gives no date beyond "the coming months" and no detail on which pipeline will read the line first.

3 traps in the fine print

Read the draft itself and a few things jump out that the press releases skip.

A wildcard group does not count. In robots.txt, a wildcard rule is the default for unknown bots, and the draft is blunt about it: a User-Agent: * group "MUST NOT be interpreted as matching the policy token." So a site that blocks everything with a wildcard has, under this mechanism, expressed no training preference at all. Silence is not refusal here.

A partial block can read backwards. The analysis site SXZ.io ran the draft's lookup rules through three robots.txt parsers and found something uncomfortable. Because RFC 9309 treats an unmatched path as allowed, a group that disallows only one folder expresses a preference against training there, and a preference in favour of training on every other path. An empty group is a preference in favour of everything. There is no way to write "no here, no opinion elsewhere." Block the root first, then list the exceptions you actually want to allow.

It is not a crawl instruction and it is not retroactive. The draft calls usage preferences and traffic rules "independent and orthogonal," and says the token "is not a directive for crawling behavior." Bots may still fetch your page. The preference that applies is the one in the file at the moment the page was retrieved, so nothing already collected changes. Private licensing deals can override it too.

One more thing worth knowing: this draft is not the only IETF proposal in play. The working group's own attachment draft, draft-ietf-aipref-attach-05, uses a Content-Usage rule placed inside a crawler's group, which behaves differently on wildcards and on blocked paths. Two mechanisms, one vocabulary, and the syntax could still move. Write the token now if you like, but don't rebuild your site around it.

The other Friday story: a UK probe into Meta

While four labs were promising a voluntary opt-out, Britain's Competition and Markets Authority opened a formal investigation into Meta's commercial terms, the ones that would let it crawl any site linked to a business account and use that content for its own products, AI included, with no way to say no.

The terms were due on October 30. Meta pushed them to November 20 after talks with the regulator, then paused the rollout on October 28. Chief executive Sarah Cardell said the CMA had "heard concerns from news publishers and other businesses and are acting swiftly." Meta's line: "There are no grounds for a competition investigation."

Put the two stories side by side and you get the shape of the year. One camp offers publishers a line in a text file that it promises to read next year. The other is being investigated for writing permission into a contract you can't refuse. Meta said it liked the robots.txt idea without saying it would honor it, and its paused terms say nothing about robots.txt at all, a gap the Publishers Association flagged.

The pressure is not only British. A Norwegian court temporarily blocked Meta from enforcing the new user terms this week after 10 media outlets challenged them, and the Paris appeals court refused on Thursday to overturn an order that Meta negotiate in good faith with French publishers. More than 340 news sites, including the Guardian and the New York Times, have blocked Internet Archive crawlers since May.

Why publishers are not celebrating

Here is the number that explains the mood. Of the top 1,000 websites that publish a robots.txt file, 87% still allow GPTBot, according to a September crawl by StackScan covering 90.8 million files. Blocking is a minority sport, and much of it happens by default: one Cloudflare setting writes a block list for eight crawlers, and that managed file sits behind 8.6 million sites.

Where sites do push back, they push back with firewalls, not text files. Cloudflare's network answered 13.47% of AI bot requests with a 403 in the third quarter, up from 5.98% a year earlier. People stopped asking nicely and started closing the door.

They have reasons. Anthropic's crawlers fetched 988 pages for every visitor they referred back in that same quarter. OpenAI's ratio was 309 to 1. Google's was 4.9 to 1. Publishers read those numbers as an invoice they never agreed to. Look, I'm not saying every crawler is a thief, but the ratio tells you who is paying for the party.

The deeper problem is that training is not the wound. Answers are. Ahrefs measured a 58% drop in click-through rates for top-ranking pages where AI Overviews appear, and Chartbeat tracked a 33% fall in Google referrals across 2,500 news sites in 2025. A token that only governs training leaves all of that untouched. Microsoft is the only one of the four offering a grounding switch, and it told you plainly that using it will cost you visibility.

Even the people who helped write the draft aren't calling it finished. Nate Hake of Travel Lemming, thanked in the acknowledgments, posted that "it's not actually done at the ietf level" and that "some significant areas of dispute remain." His specific complaint: the preferences don't apply to anything a user does, which leaves artists and creators exposed to AI copycats. Thurrott's read was blunter. The change arrives "years after all of them stole content at a scale never before seen, making the change largely symbolic."

What to do with your robots.txt this week

Adding the line costs nothing and breaks nothing, because RFC 9309 tells parsers to skip groups for agents they don't recognise. Do it properly, though, or you'll express the opposite of what you meant. Start with the root block, list exceptions, keep every old crawler rule in place, and don't fold the token into a crawler's group unless you also want that crawler blocked. Here is the shape the draft uses: User-Agent: __usage_ai-training, then Disallow: /, then Allow: /press-kit/ for the folder you want open.

Then decide about grounding separately, because that is the setting with a real traffic cost. If you want to stay out of training but keep appearing in Gemini answers, you'll need an explicit Google-Extended allow line. Google's fallback treats a bare token opt-out as a full Google-Extended block, and that control also governs Gemini app grounding. That's the trap. If you use Bing's legacy tags, put the March 1 migration in your calendar now rather than in February.

Finally, verify. Publishing a preference is not the same as a provider reading it. Check your server logs against each company's stated date, starting with Anthropic, then Microsoft on February 1. Until then, the per-crawler rules are still the only thing actually doing the work.

🎯 Key Takeaways

• The shared token is a genuine simplification, and a voluntary one. It obliges nobody who hasn't signed up.
• Only Microsoft has given a hard date, February 1, that it can be held to.
• Training-only scope means the traffic problem publishers actually have is untouched.
• The wildcard rule and the partial-block quirk mean a careless edit can read as permission.
• Keep your old crawler blocks. The token adds a preference, it doesn't replace one.

FAQ

No. The draft is explicit that it is not a directive for crawling, and all four companies describe it as a preference about use. If you want a crawler off your server, keep the per-crawler group for GPTBot, ClaudeBot or Bingbot. The token only speaks to what happens after a page has been fetched.

No. The draft ties the preference to the robots.txt file in force when a page was retrieved, so it looks forward only. Nothing in those announcements covers deletion, dataset scrubbing or models that already shipped. That question sits with courts and licensing deals.

Probably yes, at least for a while. Every company said its existing controls keep working alongside the token, and Google's documentation gives an explicit Google-Extended rule priority over the shared signal. Keeping both costs a few lines and covers the transition.

Meta's communications head Andy Stone called a common signal "an important step" and said the company looks forward to working on implementation, without promising to honor it. Days earlier Meta paused commercial terms that would have given it blanket crawling permission. Its interest in a voluntary signal, while fighting a regulator over contractual consent, is easy to read.

📚 Sources