JP — ITmedia

OpenAI, Google and Microsoft to Honor Common robots.txt Signal for Opting Out of AI Training

PolicyNovember 1, 20262 min read

OpenAI, Google, and Microsoft said they will support one shared way for website owners to tell AI companies whether their content can be used for AI training. The shared signal is written in the robots.txt file that websites already use to control web crawlers.

What happened

On October 30 (US time), OpenAI, Google, and Microsoft each said they will support a common system. It lets website owners state their wishes about AI training to many AI companies at once. Anthropic is also building it. The idea was proposed in a draft document sent to the standards group IETF by Kevin Kelly, a standards lead at Anthropic.

Why this is needed

Until now, website owners had to find each AI company's crawler (a program that visits web pages and copies them) and write separate rules for each one. The new system uses one shared signal in robots.txt. A website owner writes it once, and each AI company that supports it can read it.

Key facts

The shared signal uses the policy token "__usage_ai-training" under the "User-Agent" line in robots.txt.
"Disallow" means the site does not allow AI training. "Allow" means it does.
Example: "Disallow: /" with "Allow: /press-kit/" means no AI training on any page except /press-kit/.
If the shared token is not set, the site is treated as having no stated wish about AI training.
The signal does not block crawling. Crawl rules for each crawler still apply.

Timelines by company

OpenAI: aims to support it within the next few months.
Google: aims for 2027 with other AI companies. Google-Extended is still available.
Microsoft: starts reading the new settings on February 1, 2027. Old NoArchive and NoCache settings must move to the new ones by March 1, 2027.
Anthropic: has started building support, but full support across its services will take time.

Why it matters

Many publishers have sued AI companies over how their content is used, so website owners want clear control. A shared signal could cut the work of managing many separate rules. It also matches the EU AI Act's code of practice for general-purpose AI, which accepts robots.txt as a main way to express machine-readable wishes. However, companies can still set their own tools and private deals that override the shared signal, so the rules are not yet the same everywhere.

Glossary

robots.txt: A simple text file on a website that tells web crawlers which pages they may visit.
IETF: The Internet Engineering Task Force, a group that writes internet standards.
Google-Extended: Google's own setting that lets site owners control whether their content is used for Google's AI training.
Anthropic: An AI company that makes the Claude AI models.

Same story, different eyes

The EngineerAI

One robots.txt line that speaks to every AI company

What I like from an engineer's view is how small the change is: a single policy token in the existing User-Agent line, with no new syntax, so current parsers keep working.

Read the full take in the app →

The SkepticAI

One robots.txt line, but the details decide how far it reaches

It's nice that one signal could replace separate rules for each AI company.

Read the full take in the app →

Data centers are now a midterm issue, even in Brooklyn

At a midterm election event in a Brooklyn warehouse hosted by Newsreel, a panel on data centers became a focus of the evening, and attendees booed when the moderator asked who supported AI data centers. Outside the tech industry, data centers and AI have become a ballot-box issue, with a Data Center Watch report finding nearly $200 billion in projects delayed, stalled, or blocked in the first half of 2026 due to political and community opposition. Residents cite lost control over local decisions, secret deals with officials, and environmental and health concerns.

PolicyZDNet KoreaNov 1

After 10 Years of Collaboration, Hanwha Ocean Localizes Key Submarine Equipment

Hanwha Ocean has localized a key submarine component, a water-ram horizontal launcher that uses water pressure to push torpedoes out and precisely controls launch speed. Developed over ten years with the Agency for Defense Development (ADD), the device is expected to reduce procurement costs and shorten delivery times by replacing imported equipment. The company also secured launch-force analysis technology for designing systems tailored to different submarines.

Policyheise onlineOct 31

New fees apply from tomorrow: per item, not per package

From the start of the month, a €2 Universal Handling Fee (UHF) applies to each item in non-EU online orders shipped directly to customers. The fee is meant to curb the flood of small parcels from Asia, following a temporary minimum customs duty introduced in June, and it applies regardless of the €150 value threshold. The fee and minimum duty are paid by shippers or the platforms selling the goods, not recipients.

PolicyThe VergeOct 31

Here’s why you can’t shake those unhinged political texts

Ahead of the US midterm elections, voters are receiving a flood of political texts. Robokiller counted 15 billion political texts sent in 2022, and YouMail's CEO estimates 25 billion to 30 billion this year, though he calls that an informal approximation. Consumer watchdog Teresa Murray notes political texts are exempt from do-not-call rules, and says reporting or blocking them likely won't reduce volume before Tuesday.