Archive — US — techcrunch
AI crawlers cause Wikimedia Commons bandwidth demands to surge 50%
Wikimedia Foundation reports a 50% surge in bandwidth demands for downloads from Wikimedia Commons, attributed to AI crawlers rather than human users. These crawlers are causing high costs and resource strains, prompting efforts to block them and potentially threaten open internet availability.
Key points
- Bandwidth consumption surged by 50% since January 2024 due to automated AI scrapers.
- 65% of the most resource-intensive traffic comes from bots, despite these bots contributing only 35% of overall pageviews.
- Bots tend to 'bulk read' less popular pages, increasing costs as this content is served from core datacenters.
- Wikimedia is diverting resources to block crawlers to maintain user experience.
- The issue reflects a broader trend impacting open-source infrastructure, with developers and companies responding to the surge in bot traffic.
Why it matters
- The growing use of AI crawlers poses significant challenges for public internet resources.
- Similar complaints from developers about increased bandwidth demands highlight the impact on open-source projects.
- Response mechanisms like Cloudflare's AI Labyrinth indicate efforts by tech companies to mitigate crawler effects.
The situation at Wikimedia Commons illustrates the rising threat AI crawlers pose to the sustainability of open knowledge platforms, prompting urgent defensive measures that risk limiting access to information online.