Artificial Analysis, an independent AI benchmarking organization, released a major overhaul to its Intelligence Index on Monday, fundamentally changing how the industry measures artificial intelligence progress. The new Intelligence Index v4.0 incorporates 10 evaluations spanning agents, coding, scientific reasoning, and general knowledge, moving away from traditional benchmarks that the organization deemed obsolete.

The organization removed three staple benchmarks – MMLU-Pro, AIME 2025, and LiveCodeBench – which have been widely cited by AI companies in their marketing materials. These were replaced with evaluations designed to measure whether AI systems can complete tasks mirroring real-world work that people are paid to do. This shift reflects a growing concern that existing benchmarks focus too heavily on recall and not enough on practical application.

The Intelligence Index serves as a closely watched ranking system for AI models, influencing both developers and enterprise buyers. The overhaul signifies a critical adjustment in how AI progress is evaluated. Instead of prioritizing performance on standardized tests, the new index emphasizes the economic utility of AI systems. This change comes as AI models rapidly improve, rendering older benchmarks less effective at differentiating capabilities.

"This index shift reflects a broader transition: intelligence is being measured less by recall and more by economically useful action," observed Aravind Sundar, a researcher who responded to the announcement. This perspective highlights the evolving understanding of AI intelligence, moving beyond simple knowledge retrieval towards problem-solving and practical application.

The implications of this change are significant for the AI industry. Companies may need to rethink their marketing strategies, focusing less on raw benchmark scores and more on demonstrating real-world capabilities. Enterprise buyers will likely place greater emphasis on evaluations that reflect their specific needs and use cases. The updated index aims to provide a more accurate and relevant assessment of AI systems, guiding development and adoption in a more practical direction. The new index is available immediately, and Artificial Analysis plans to continue refining the evaluations based on ongoing developments in the field.