US — The Verge
The biggest names in AI are backing a new way to opt out of scraping
AI1 min read
OpenAI, Google, Microsoft, and Anthropic say they will support a new shared signal that lets websites block AI companies from scraping their content. The signal is a single setting that works across many AI firms.
A shared way to say no to AI scraping
Four big AI companies say they will follow a new standard. It lets publishers set their AI training rules once, instead of telling each AI company separately. The idea was proposed by Kevin Kelley, a technical lead for standards at Anthropic, through the Internet Engineering Task Force (IETF), a group that writes internet rules. The setting lives in a website's robots.txt file, a simple text file that tells web crawlers (automatic programs that read websites) what they may visit.
Key facts
- Four companies say they will adopt the signal: OpenAI, Google, Microsoft, and Anthropic.
- Anthropic has already adopted the signal.
- Google and Microsoft plan to support it next year.
- OpenAI says it will add support in the "coming months."
- Microsoft will offer finer controls, so publishers can opt out of AI training or AI grounding separately.
Why publishers are worried
Many publishers say AI crawlers keep visiting their sites and copying content into AI models. Some groups have tried other fixes. The Really Simple Licensing (RSL) standard lets publishers charge crawlers that visit their sites. Cloudflare, a company that helps run websites, blocks AI crawlers by default.
Why it matters
This could make it much easier for website owners to control how their work is used by AI. Instead of managing many settings, they can use one signal across several companies. Still, the signal only works if companies agree to follow it. Microsoft also notes that websites will still show up in Bing search results, no matter what preferences a site sets.
What to watch
- Whether Google, Microsoft, and OpenAI actually add support on their promised timelines.
- How well Microsoft's finer controls for AI training and AI grounding work in practice.
- Whether other AI companies also agree to respect the shared signal.
- How the IETF turns this idea into an official internet standard.
Glossary
- robots.txt
- A plain text file on a website that tells web crawlers which parts of the site they may or may not visit.
- Internet Engineering Task Force (IETF)
- A group of engineers that writes and shares rules for how the internet works.
- AI grounding
- Pointing an AI model to a trusted source of facts to help it give more accurate answers and fewer made-up ones.
- Really Simple Licensing (RSL)
- A standard that lets publishers charge AI crawlers that visit their sites.
- Cloudflare
- A company that provides services to help websites run safely and fast, and which blocks AI crawlers by default.
- Anthropic
- An AI company that makes the Claude AI models and has already adopted the new signal.