Archive — US — techcrunch
People are using Super Mario to benchmark AI now
Researchers from UC San Diego's Hao AI Lab used Super Mario Bros. to benchmark AI, finding that Anthropic's Claude 3.7 outperformed other models, including OpenAI’s GPT-4o. The study highlighted the challenges for reasoning models in real-time gameplay due to slower decision-making, sparking a discourse on the relevance of gaming benchmarks in evaluating AI capabilities.
Key points
- Hao AI Lab tested AI performance in Super Mario Bros., using an emulator and the GamingAgent framework.
- Anthropic’s Claude 3.7 outperformed other models, including Claude 3.5, Google’s Gemini 1.5 Pro, and OpenAI’s GPT-4o.
- Models were fed instructions to navigate the game, generating actions via Python code.
- Reasoning models performed worse in real-time gameplay, as they take longer to decide on actions.
- This research highlights concerns about the validity of gaming benchmarks for AI evaluation.
Why it matters
- AI gaming benchmarks have a historical precedent but are now questioned for their connection to real-world AI capabilities.
- As of 2025, there's ongoing discussion on what constitutes meaningful metrics for evaluating AI performance, especially in dynamic environments.
- The increasing complexity of AI applications parallels the evolving nature of benchmarking practices beyond traditional metrics.
The exploration of AI in gaming like Super Mario Bros. raises critical questions about the efficacy of benchmarks, suggesting the need for more robust evaluation metrics to gauge real-world applicability.