Archive — US — techcrunch
Perplexity accused of scraping websites that explicitly blocked AI scraping
AI startup Perplexity is accused by Cloudflare of scraping content from websites that explicitly blocked AI scraping. Despite using methods to circumvent these blocks, Perplexity denies the allegations, claiming the identified bot is not theirs. This incident raises concerns about AI's impact on web content rights, as more websites employ rules to restrict scraping.
Key points
- Cloudflare accuses Perplexity of ignoring website scraping restrictions.
- Perplexity allegedly uses bot identity obscuration techniques to bypass Robots.txt.
- Cloudflare confirmed the behavior after complaints from client websites.
- Perplexity's spokesperson dismissed the allegations, claiming no content was accessed.
- Cloudflare has taken measures to block Perplexity's bots and launched a marketplace for websites to charge scrapers.
Why it matters
- The incident highlights ongoing tensions between AI data scraping and content creators' rights.
- Historically, web scraping was largely unregulated, but growing awareness is leading to preventive measures.
- By 2025, trends continue to emphasize AI’s ethical use, including consent and data ownership.
The accusations against Perplexity underscore critical challenges in AI ethics, particularly regarding content ownership and scraping practices. With increasing scrutiny on AI operations, the future of data usage will likely demand clearer standards and respect for website owners' preferences.