Archive — US — techcrunch
Anthropic used Pokémon to benchmark its newest AI model
Anthropic tested its AI model Claude 3.7 Sonnet using Pokémon Red, enabling it to play via basic memory and function calls. The model surpassed its predecessor by defeating three gym leaders and showcased its extended thinking ability. Despite being a toy benchmark, this approach continues a trend in using games for AI evaluation.
Key points
- Anthropic benchmarked Claude 3.7 Sonnet using Pokémon Red.
- The model utilized basic memory and screen input features.
- Claude 3.7 Sonnet features extended thinking, allowing it to reason through complex issues.
- Compared to Claude 3.0 Sonnet, which failed to progress, Claude 3.7 successfully battled and won badges from three gym leaders.
- Claude 3.7 performed 35,000 actions to confront the final gym leader.
- The benchmark is seen as a playful yet common practice in AI testing through games.
Why it matters
- Previously, games like Space Invaders and Chess have been important in AI development.
- The use of video games for AI benchmarks aligns with 2025 IT trends emphasizing gamification in technology.
- AI models are increasingly adopting game-playing elements to evaluate problem-solving and reasoning skills.
The use of Pokémon Red as a benchmarking tool for AI exhibits the evolving integration of gaming in AI testing, reflecting ongoing trends in technology that leverage gamified environments for enhanced problem-solving evaluations.