Introduction
Humanity’s Last Exam (HLE) stands as a pioneering benchmark crafted to assess the reasoning and deep knowledge capacities of modern AI systems. Its hallmark is an extreme valuation, serving as the latest evolution of Turing tests, a concept that has been etched in history for decades.
This article delves into the nuances of this benchmark, exploring its inception, gathering diverse expert opinions, and culminating with a consensus on its significance.
Why was it built and what does it consist of?
With the evolution of AI systems, traditional testing methods have become obsolete, as systems began achieving flawless results effortlessly. In response, the Center for AI Security, in collaboration with AI Scales and global experts, developed HLE. This benchmark was published in Nature, one of the most esteemed scientific journals, in January 2026. Its design deliberately avoids the repetitive patterns that plagued previous assessment frameworks.
HLE is an exam for cutting-edge AI systems, including language models, featuring over 2,500 expert-level questions across a hundred academic disciplines such as physics, mathematics, biology, humanities, and more. Notably, these questions cannot be answered through mere memorization or basic information retrieval. Instead, they demand complex deductive reasoning and profound understanding.
Here is an example of two of these questions:
Two examples of HLE questions. Image source: Center for AI Security
Examining the results achieved by the most advanced models, even sophisticated frontier models like GPT, Gemini, or Claude struggle, barely surpassing an overall accuracy threshold of 45 to 50%. These figures underscore the exam’s formidable difficulty. Moreover, these models often falter due to overconfidence in incorrectly answered questions.
What is the opinion of the mainstream experts on HLE?
The candid response is a lack of consensus. Opinions diverge among technology, developer, and academic circles, although there is a prevailing trend towards acknowledging HLE’s real utility. However, critical nuances persist.
Experts and the informed public do not dismiss HLE as meaningless, but they often criticize its marketing-oriented nomenclature. Broadly, three dominant opinion groups have emerged:
1. HLE is really useful and necessary
Approximately 60% of reviews align with this perspective, emphasizing a technical rationale for HLE’s current importance. Previous benchmarks like Massive Multitask Language Understanding (MMLU) have become saturated, with modern AI scoring over 90%, making it challenging to differentiate the latest models. HLE is praised for assessing whether AI is willing to admit “I don’t know” rather than fabricating answers to complex questions.
2. HLE is a distraction from real AI
About 30% of reviews espouse this skeptical view, arguing that the test does not accurately assess AI’s performance in real-world scenarios, focusing instead on esoteric academic knowledge. Some engineers jest that once AI consistently achieves over 90% on HLE, companies will inevitably create HLE 2, perpetuating a marketing cycle favoring large corporations.
3. HLE is defective
This minority view, discussed in data science forums, criticizes HLE for errors in some answers deemed correct, particularly in specialized areas like chemistry and advanced mathematics. Interestingly, it was the most advanced AI systems themselves that began identifying these benchmark errors.
Conclusion
In conclusion, while HLE’s usefulness is not disputed and many experts emphasize its importance, its name is widely regarded as marketing hyperbole. HLE is unlikely to herald the advent of a super AI or the true emergence of general artificial intelligence (AGI), a long-debated concept still more fiction than reality. Nevertheless, HLE is considered an ambitious tool for discerning which AI or company possesses the superior model in terms of memory and logical capabilities.
Ivan Palomares Carrascosa is a leader, writer, speaker, and advisor in AI, machine learning, deep learning, and LLM. He trains and guides others in leveraging AI in the real world.
Source: Here
“`
