Archive — US — techcrunch
Cloudflare’s new policy pushes AI companies to pay for publishers’ content
Cloudflare announced a new policy requiring AI companies to pay for publishers' content, effective September 15, 2026. The policy will block mixed-use web crawlers by default from accessing ad-hosting pages unless site owners adjust settings. This aims to protect intellectual property and compensate publishers for content used in AI training.
Key points
- Cloudflare sets a deadline of September 15, 2026, for AI companies to separate crawlers for traditional search from those used for AI training.
- Mixed-use crawlers will be blocked from ad-hosting pages by default unless site owners change settings.
- Most website owners desire visibility for their content, but want protections against free use by AI.
- Cloudflare highlights that Google's search engine has more access to information than other AI companies, making it harder for content to remain undiscovered.
- The policy aims to create a sustainable ecosystem as non-human traffic overtakes human traffic online.
- Cloudflare offers new tools and partnerships to help publishers monetize their content through AI crawlers.
- Initial partners for the program are Ceramic.ai and You.com, which will compensate publishers for their content.
Why it matters
- Historically, the relationship between content creators and search engines has been contentious, as publishers seek fair compensation.
- This move aligns with a growing trend of monetizing online content amid the rise of AI technologies, reflecting the 2025 IT trend of enhanced digital rights management.
- In 2022, the internet saw increased discussions on AI training ethics, emphasizing the need for transparent use of web content.
Cloudflare's policy marks a significant shift in the AI landscape, aiming to protect content creators and ensuring fair compensation for the use of their material in AI training, which could redefine digital content economics.