The AI training opt-out that publishers have been begging for is finally getting real support, and my honest reaction is relief mixed with a big eye roll. OpenAI, Google, Microsoft and Anthropic now say they'll honor one shared line in robots.txt that tells their crawlers "don't train on my stuff." That's genuinely useful. It also arrives years after these same companies vacuumed up most of the open web.
Still, I run a small blog. I'd rather have one switch that works than a dozen I have to babysit. So let's talk about what actually changed and whether you should flip it on.
What just happened
On October 30, The Verge reported that the four biggest AI players are getting behind a single "policy token" for robots.txt. The idea came from Kevin Kelley, a standards lead at Anthropic, and it's being worked on inside the Internet Engineering Task Force's AI Preferences group. You can read the actual IETF draft if you enjoy that sort of thing. I do, which says a lot about my Saturday nights.
Until now, if you didn't want your posts feeding a chatbot, you had to hunt down every AI crawler by name and write a separate rule for each one. Miss one, and congrats, you're training data. OpenAI's own announcement admits that per-crawler approach was a chore for site owners, which is a fun thing to read from a company that benefited from that chore being annoying.
How the AI training opt-out actually works
The clever bit is that nobody invented new syntax. You just add a fake "user agent" group to robots.txt using a special token. According to Engadget and the draft itself, it looks like this:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/That example says "no AI training on anything, except my press kit folder." You can carve out whatever paths you like using the same Allow and Disallow rules you already know.
Because it reuses the normal robots.txt format, old parsers that don't understand the token just skip it like any unknown bot. Nothing breaks. I love boring, backward-compatible engineering, and this is exactly that.
It's about training, not crawling
This is the part people will get wrong. The token is a preference about using your content for AI training. It isn't a "stay off my site" command. The draft is explicit that it's not a crawling directive, and if you want to actually stop a bot from fetching pages, you still need its specific rules.
Google says it'll treat the signal as a stop for Google-Extended, the agent that gathers material for Gemini training. Your regular search ranking isn't supposed to be touched. Microsoft says your site stays in Bing results no matter what you choose, and it plans finer controls that separate training from "grounding," which is when an AI tool pulls in a trusted source to back up an answer.
Who supports it, and when
Here's the timeline as reported by Engadget and CNA:
- Anthropic has already started honoring it.
- OpenAI says "in the coming months."
- Microsoft starts on February 1, 2027.
- Google says sometime in early 2027.
Meta is the awkward guest at this party. Its communications head Andy Stone said a common signal is an important step, but per The Verge he didn't actually say Meta would honor it. Supporting an idea and following it are two very different things, as anyone who's ever bought a gym membership knows.
The catches nobody should ignore
First, it doesn't rewind time. If a crawler already grabbed your archive, that content is already sitting in someone's dataset. This only affects future scraping and future training runs.
Second, it's opt-out, not opt-in. Microsoft said scraping stays allowed by default until a publisher changes the setting. So the millions of small sites whose owners never read tech news (or never touch robots.txt) stay fair game. That default is doing a lot of quiet work for the AI companies.
Third, it's voluntary. OpenAI calls it a "voluntary effort" right in its post. The shady scrapers that ignore robots.txt today aren't going to suddenly develop manners because the polite companies signed a pledge.
And then there's the trade-off that Microsoft and Google both flagged: opting out could make your content less likely to surface inside AI apps. That's the quiet threat baked into all of this. Block training, and maybe you vanish from the place where more and more people ask their questions.
Context matters here too. The New York Times has been suing OpenAI and Microsoft since December 2023 over training on its articles. Cloudflare has been blocking AI bots by default on new customer sites since July 2025. There's even a standard called Really Simple Licensing that lets publishers charge bots. So this shared token didn't come from nowhere. It came from pressure, and I suspect the companies would rather offer a tidy opt-out than face more lawsuits and paywalls.
Should you care?
If you publish anything online, yes, at least a little.
For big news sites, this is a cleaner way to say no, and their lawyers will probably love having a clear, standard signal on record. For people like me who run a modest tech blog, the decision is murkier. I want readers. A chunk of discovery now happens inside AI answers. Locking my posts away from training might protect my work, or it might just make me invisible in yet another place.
My take? I'd add the token for anything that's truly yours and hard to replace, like original reviews and tutorials. I'd probably leave a press-style "about" page open. And I wouldn't panic about search, since Google and Microsoft both say regular indexing isn't affected.
If you're on Blogger like me, there's a custom robots.txt option tucked inside the blog's settings under crawlers and indexing. Just be careful in there. One bad Disallow line can hide your whole site from search, and that's a far worse problem than a chatbot learning your writing style.
The bigger point is that a single standard beats a messy pile of per-company rules. I'll give the industry credit for that. I just won't pretend it's generosity when it showed up this late, after the buffet was already eaten.
FAQ
Does the AI training opt-out remove my content from existing AI models?
No. It only applies to future scraping and training. Anything already collected stays collected.
Will adding the token hurt my Google ranking?
Google says the signal maps to Google-Extended for Gemini training, not to regular search crawling, so normal search visibility shouldn't change. Microsoft says the same for Bing.
Do I still need my old robots.txt rules for AI bots?
For now, keeping them is the safe move. OpenAI says its existing crawler controls still apply, and not every company has switched the new signal on yet.
Will every AI company respect it?
Only the ones that choose to. It's voluntary, and Meta hasn't committed to honoring it.
So here's my question for you: are you adding the new token to your site, or does staying visible inside AI answers matter more to you than keeping your work out of training data?
0 Comments