Robots.txt gets a common signal for AI training

In this article you will see:

• What the new robots.txt identifier communicates.
• When each company plans to recognize it.
• Why the IETF draft does not block crawlers.
• What site teams can prepare now.

The new robots.txt token expresses preference about AI training

The proposed control uses a special identifier in the User-agent field of robots.txt. The token __usage_ai-training separates the preference about training from the crawling rules used by other bots.

The Google Search Central describes this shared control as a way to express preferences about training.

An example instruction to not allow this use would be:

User-agent: __usage_ai-training
Disallow: /

This syntax communicates a preference to systems that recognize the token. It does not by itself change a page's visibility in search, it does not remove content from the index and it does not prevent a crawler from ignoring the instruction.

The draft also allows organizing rules by path. A site can express different preferences for public areas and documentation.

Content will only stay out of training if services adopt the specification and respect that choice.

The proposal tries to replace a collection of specific bot names with a common signal.

Today, each company can interpret directives and identifiers in its own way. This makes it hard to know if a preference will be understood.

Google and Microsoft announced different timelines

The Google Search Central published the plan on October 30 and intends to support the signal in 2027. The announcement does not give a specific day for deployment.

Google also differentiates the new token from Google-Extended, an existing control for the use of content in grounded responses in the Gemini apps.

Microsoft said it will start recognizing preferences on February 1, 2027. The company says the choice does not change page eligibility in Bing search.

Microsoft also plans separate controls for grounding, when a response queries pages to compose current information.

The difference between training and grounding matters for those who publish documentation. Microsoft details the separation in the controls announcement.

Training uses examples to change a model's behavior. Grounding queries content to inform a generated response, without necessarily incorporating it into the model's parameters.

The IETF draft does not create a mandatory block

The signal is described in an Internet-Draft from the IETF, a working document that can receive comments and change. It is not yet an approved RFC nor a mandatory rule for all crawlers.

The IETF draft describes the token as a use preference, not as an order to prevent access to content.

robots.txt is public and voluntary. It helps cooperative services understand rules, but it does not work as authentication or firewall.

Therefore, the announcement does not mean a site will be technically protected against copying or collection. To restrict access to private files, authentication, authorization and server side controls remain necessary.

The change should be read as a more standardized way to communicate intent. Effective protection depends on each service interpreting the rule, declaring that it respects it and applying the behavior in its crawler.

What site teams can prepare now

Before changing the production file, confirm which preference the business wants to express. A page can stay public for research and discovery, but have a different rule for training or for grounding in AI generated responses.

Validate robots.txt in a test environment and keep the current rules for search crawlers that need to access the site.

A broad block can remove pages from traditional results without stopping collection by services that do not follow the new token.

After publishing a rule, compare the file with the server access logs. Logs help identify which bots visit the pages, but they do not prove on their own that a service used or stopped using the content for training.

This discussion connects to changes in traffic in conversational search. See the report on AI Mode and Reddit's concern about traffic to the web.

Google also expands monetization of search. Read the news about AI Mode ads reaching new countries.

The next step is to track each vendor's dates and check the documentation for the crawlers the site actually receives. Until adoption takes effect, do not assume that inserting the token already changed a company's behavior.