Archive — US — techcrunch
Cloudflare launches a marketplace that lets websites charge AI bots for scraping
Cloudflare has launched a marketplace named 'Pay per Crawl' to allow website owners to charge AI crawlers for accessing their content. This initiative aims to give publishers more control and provide a potential revenue model amidst rising AI usage that affects their traditional business models. The marketplace is currently in private beta and focuses on a permissions-based approach for content scraping.
Key points
- Cloudflare's new marketplace allows website owners to charge AI crawlers a set fee for scraping their content.
- The service is currently in private beta, providing flexibility for website owners to either allow, block, or charge for AI access.
- Cloudflare aims to offer a sustainable business model for publishers in light of declining Google search traffic and the rise of AI chatbots.
- Publishers historically relied on Google for traffic, but AI crawlers scrape significantly more than they generate referrals.
- Large publishers and organizations support Cloudflare's initiative to block AI crawlers by default to protect their content.
Why it matters
- AI content scraping has become a controversial issue, with many publishers filing lawsuits against companies using their articles for AI training.
- The rise of AI has prompted discussions about new revenue models for publishers, as they adapt to a changing digital landscape.
- Cloudflare's marketplace could shape future standards for content sharing between publishers and AI developers, reflecting broader trends in digital rights management.
Cloudflare's Pay per Crawl marketplace represents a significant shift in the relationship between digital content providers and AI, potentially reshaping how publishers monetize their content while addressing the challenges posed by AI scraping.