One robots.txt Line Now Opts You Out of AI Training at OpenAI, Google, Microsoft, and Anthropic

On October 30, 2026, OpenAI, Google, Microsoft, and Anthropic announced support for a single shared signal that lets a website owner express a preference about AI training in one place. The signal is a policy token in robots.txt, the plain text file that websites use to give automated visitors instructions about their content. The token is __usage_ai-training, with two leading underscores. A site owner writes a robots.txt group keyed by that token, and each participating lab reads that group as a statement about one purpose only: generative AI training. That is the whole announcement in one paragraph, and it is genuinely useful, because it replaces a growing list of per company rules with one shared preference. The announcement matters most to publishers, bloggers, businesses, and anyone whose website content is the product. Until now, a site that wanted to opt out of training had to identify each lab's crawlers, learn each lab's control, and maintain separate rules that could drift out of date as crawlers changed names or new labs appeared. A single shared token collapses that maintenance into one group, and it makes the preference readable in the same way by every participating provider. That is why The Verge, Search Engine Land, and the labs' own posts all framed October 30 as a first coordinated step, not as a finished solution. The step is real, but it is a step, and the rest of this article is about the gaps that a careful site owner must not ignore. This article covers the exact line to add, what each lab has actually committed to and when, what the token does not do, and the safe way to add it without breaking search. It does not claim that adding the line today protects your content everywhere, because it does not, and it does not claim that the token is a law or a finished standard, because it is neither.

What is the exact line, and how do you write it?

The token comes from an IETF Internet-Draft titled Machine-Readable Training Preference Signal, authored by Kevin Kelley, a technical lead for standards at Anthropic, and dated October 30, 2026, with an expiry of May 3, 2027 unless it is updated. It was submitted to the IETF's AI Preferences working group, known as AIPREF, which is chaired by Cloudflare's Mark Nottingham and Cisco's Suresh Krishnan. That history matters for one reason: the document is an individual Internet-Draft, not a formal IETF standard, and the IETF datatracker itself notes that an individual draft is not endorsed by the IETF and has no formal standing in the standards process. The labs have chosen to honor it voluntarily. They are not bound by a standard, and a lab that changes its mind is not breaking a standard. It is breaking a public commitment, which is a different kind of pressure, and it is worth naming that difference honestly. The syntax uses ordinary robots.txt groups, with no new directives and no new grammar. A site that wants to opt out of training for its whole site writes: User-Agent: __usage_ai-training on one line, then Disallow: / on the next line. A site that wants to opt out everywhere except one section, for example a press kit it is happy to have used for training, writes: User-Agent: __usage_ai-training, Disallow: /, Allow: /press-kit/. That second example is the draft's own example, and it shows the useful part of the design: the preference can be path specific, so a site does not have to choose between all in and all out. Specific expressions take precedence over general ones, and a lab can treat its own more specific token, such as Google-Extended for Google, as more specific than the shared token. That precedence rule is how a site can set a broad shared choice and then fine tune it for one provider, without rewriting the whole file. Three syntax traps cause most mistakes, and all three are stated in the draft. First, the token is a policy token, not a real crawler. There is no bot named __usage_ai-training visiting your site. It is a label that participating providers read as a preference. Second, a wildcard group does not match the token. A file that only has User-Agent: * with Disallow: / has not expressed any training preference under this draft, even if the site owner intended it to. The draft states that wildcard groups must not be interpreted as matching the policy token, and that if no group in a file contains the token, no preference has been expressed at all. You must write the token group explicitly, even if your file already blocks everything for a wildcard. Third, the token group is not a crawl block. Disallow: / under this token means do not use this content for training. It does not mean do not visit, do not index, or do not show this page in search. The draft states that the policy token is not a directive for crawling behavior, and that content use preferences and crawler traffic management are independent. A site that confuses the two can either fail to protect its training preference, or accidentally block search, depending on which direction the confusion runs.

What has each lab actually committed to, and when?

The four commitments are not identical, and the dates are the most practical fact in the whole announcement. A signal added today may be read by one lab months before another lab reads it, and a site owner who assumes uniform protection on October 31 will be wrong for most of 2027. The table below states each commitment as reported from the labs' own October 30 posts and their crawler documentation, with the careful wording each lab used.

What does the token not do?

The token does not block crawling, does not block search indexing, and does not stop a page from appearing in AI written search answers that quote or ground on live web content. That last point is the one most likely to disappoint a publisher who adds the line and then sees their article quoted in an AI answer the next week. Training and grounding are different uses, and this token covers training only. A site that wants to control grounding, search AI features, or snippet use needs the separate controls for those uses, where they exist, and must accept that some AI answer uses have no equivalent opt out at all. The token also does not remove content already collected, already in a dataset, or already trained into a model. It is a forward looking preference for future training runs, not a deletion request and not a way to unlearn a model. A publisher who opts out today is protecting future training, not reversing past ingestion, and any article that promises otherwise is overstating the announcement. The token also does not bind labs that did not join the announcement. Meta is the clearest example: Meta welcomed a common signal, according to reporting, but did not confirm it will honor this token, and Meta is not a party to the four lab announcement. Smaller labs, scrapers, data brokers, and Common Crawl style datasets are also outside the commitment, unless they separately choose to honor it. A site that only adds the shared token and deletes every other rule will have weaker protection against non participants than a site that keeps its per lab rules, its rate limits, and its access controls. The shared token simplifies the four lab case. It does not replace a full bot management plan. There is also a legal caution. The token is a voluntary preference signal, not a licence, not a contract, and not a copyright enforcement tool on its own. It states a preference in a machine readable form, and participating labs have publicly committed to honor it. Whether a lab's failure to honor it would be a legal violation depends on the jurisdiction, the contract, and the facts, and this article does not give legal advice on that question. Publishers with a real dispute, a licensing deal, or a large archive should take legal advice before relying on a robots.txt line as their only protection, and should keep records of when they added the token, what the file contained, and which lab posts they relied on. A dated copy of your robots.txt file is cheap evidence, and it is the kind of evidence that matters if a commitment is ever tested.

How should you add it safely, without breaking search?

Add the shared token group to your existing robots.txt file, do not replace the file with only that group, and do not put the token group inside a group for a real crawler. Keep your existing groups for Googlebot, GPTBot, OAI-SearchBot, Google-Extended, ClaudeBot, and any other crawler you already control, because those rules are the ones actually enforced today. Add the shared token as its own group, at a place in the file where it is easy to find and audit, and add a comment above it with the date you added it, so a future editor knows why it is there. A comment in robots.txt starts with a hash, and it does not affect parsing, but it does affect whether the next person who edits the file understands the intent and leaves the group alone. Test the file after you edit it. Fetch your robots.txt file in a browser and confirm the token group is present, spelled exactly as __usage_ai-training with two leading underscores, and not mangled by a plugin, a redirect, or a caching layer. Check that you have not accidentally put Disallow: / under User-Agent: * in a way that blocks search crawlers, and check that your search groups still allow the crawlers you want. Use Google Search Console's robots.txt tester or an equivalent validator if your platform offers one, and check your search traffic for a week after the change, not just on the day you make it. A robots.txt mistake often shows up as a slow search decline, not as an instant error, so a one day check is not enough. Decide the path question before you publish the group. Do you want to opt out for the whole site, or only for some sections? A news publisher may want to opt out for reporting but allow a press kit, a documentation site may want to allow training use for public docs but opt out for a members area, and a business may want to opt out for blog content but allow product pages that benefit from AI discovery. The token supports path specific choices through Allow lines, so use them deliberately, and write down the decision in one sentence for your team: this site opts out of training for these paths, and allows it for these paths, for these reasons. That sentence is what stops a future redesign from silently deleting the group or copying a template file that lacks it. Finally, set calendar reminders for the rollout dates, not just for today. Anthropic has started, OpenAI says coming months with a further update promised, Microsoft starts February 1, 2027 with a March 1 deadline for old tags, and Google aims for early 2027. Check each lab's implementation post when it appears, and retest your file at each date. A preference that is added in October 2026 and never checked again will be correct only if every lab implements it exactly as announced, on time, and without a syntax surprise. That is a lot to assume for a draft that expires in May 2027 unless it is updated. The shared token is a genuine improvement for publishers, and it is worth adding this week, but it is worth adding as part of a maintained robots.txt file, not as a one time fix that is forgotten. Key takeaways:

• On October 30, 2026, OpenAI, Google, Microsoft, and Anthropic backed a shared robots.txt policy token, __usage_ai-training, so a site can state a training preference once for all four labs.
• The exact opt out group is User-Agent: __usage_ai-training followed by Disallow: /, with Allow lines for any sections you want to exempt.
• Anthropic has begun implementation, OpenAI aims for the coming months, Microsoft targets February 1, 2027, and Google aims for early 2027, so protection is staggered, not instant everywhere.
• The token is an individual IETF Internet-Draft, voluntary, not a finished standard, and it expires May 3, 2027 unless updated.
• The token covers generative AI training only. It does not block crawling, search indexing, or AI answers that ground on live content, and it does not remove content already collected or trained.
• A wildcard User-Agent: * group does not match the token, so you must write the token group explicitly, and you should keep existing per lab rules such as GPTBot and Google-Extended in place until each lab's implementation is confirmed.
• Meta has not committed to honor this token, and Microsoft's separate grounding control and March 1, 2027 deadline for old NoArchive and NoCache tags need separate attention. Related links: Top Use AI Blog, Best free SEO tools, YouTube AI content monetization Affiliate disclosure: No affiliate link is used in this article. Links to OpenAI, Google, Microsoft, Anthropic, the IETF, and TopUseAI posts earn no commission. Sources: OpenAI, Making it easier for websites to express AI training preferences, October 30, 2026; Google Search Central blog, shared AI training control announcement by John Mueller, October 30, 2026; Microsoft Bing announcement, October 30, 2026; IETF Internet-Draft draft-kelley-aipref-training-preference-signal, Machine-Readable Training Preference Signal, Kevin Kelley, October 30, 2026; reporting by The Verge, Search Engine Land, and Search Engine Roundtable on October 30, 2026, as summarised October 30 to 31, 2026.