Archive — US — techcrunch
Some people are defending Perplexity after Cloudflare ‘named and shamed’ it
Cloudflare accused AI search engine Perplexity of stealthily scraping websites despite specific blocks. In response, many defended Perplexity, arguing that it was akin to human browsing. The incident highlights ongoing debates over AI web crawling ethics and the intersection of AI and user rights as bot activity becomes a significant online presence.
Key points
- Cloudflare accused Perplexity of scraping sites and ignoring robots.txt blocks.
- Defenders argue Perplexity's actions resemble human browsing rights.
- Cloudflare's CEO compared Perplexity's methods to malicious actions, calling for punitive measures.
- Perplexity claims the scraping behavior was from a third-party service.
- Debate centers on the difference between AI agent requests and human requests.
- Bot activity is increasing, with AI traffic surpassing human traffic on the internet.
Why it matters
- This controversy reflects larger discussions about AI ethics, especially as AI assistants become more prevalent.
- The use of bots is transforming internet dynamics, with predictions suggesting a decline in traditional search queries due to LLM traffic.
- The rise of AI technology calls for new standards to balance openness and protection on the web, including Web Bot Auth.
The clash between Cloudflare and Perplexity illustrates the ongoing struggle to define the boundaries of AI activity online, emphasizing the need for new standards as bots increasingly dominate web interactions.