EXPLORINEER
en

Archive — US — techcrunch

A new, challenging AGI test stumps most AI models

The Arc Prize Foundation introduced ARC-AGI-2, a new test for AI models that challenges their general intelligence, yielding low scores (1%-1.3%) among leading models. Designed to assess efficiency and adaptability, the test distinguishes itself from ARC-AGI-1, which relied heavily on computing power. The Foundation seeks to refine AI evaluation with a new contest aiming for 85% accuracy at low costs.

Key points

  • The Arc Prize Foundation announced a new AI assessment called ARC-AGI-2.
  • Most AI models, including top performers, scored only 1%-1.3% on the test.
  • The test consists of puzzle-like problems requiring models to identify visual patterns.
  • It evaluates not only problem-solving skills but also the efficiency of acquiring new skills.
  • Greg Kamradt emphasized the importance of efficiency alongside problem-solving abilities.
  • The previous test, ARC-AGI-1, was in use for five years until it was surpassed in December 2024.
  • OpenAI's o3 model achieved high scores on ARC-AGI-1 but performed poorly on ARC-AGI-2.
  • The Foundation launched a contest for developers aiming for 85% accuracy on ARC-AGI-2 at minimal cost.

Why it matters

  • ARC-AGI-2 addresses the industry's call for better benchmarks measuring AI capabilities.
  • The test's introduction aligns with ongoing discussions about AI's evaluation methods after the release of models like OpenAI's o3.
  • There is a growing concern regarding the sustainability of AI development costs illustrated by the performance of modern AI models.

ARC-AGI-2 sets a new standard for evaluating AI intelligence, emphasizing efficiency and adaptability, reflecting the evolving demands of the tech industry for robust benchmarks in AI performance.