AI labs have staked their reputations on model benchmarks—evaluation systems intended to be neutral and accurate, designed to determine whether one AI product outperforms another.
However, as Irene Solaiman, Chief Policy Officer at Hugging Face, noted during a discussion at last week’s TechCrunch Disrupt conference, these evaluation methods are complex and still evolving. If the industry intends to rely on such benchmarks in the future, the testing process must mature into a more standardized, scientific framework.
Given this, can we trust the results currently being published?
“Trust is a very sensitive word,” Solaiman said. “When it comes to whether we can trust the results, a key factor is: can you reproduce the evaluation? Do you have the necessary access to the model itself? This is where ‘open weights’ play a crucial role.”
The reproducibility of model evaluations has long been one of the most challenging issues in benchmarking—and a persistent subject of debate within the industry.
Beyond the validity of any individual test, there is the further question of how one test compares to another.
“These are an evolving science,” Solaiman remarked. “We don’t yet know how to make meaningful comparisons between them.”
She added, “Their efficiency would soon decline, and we have yet to establish a sufficiently robust ecosystem for evaluation. While there is a growing desire to build an ’embedded’ evaluation ecosystem, the criteria for determining who qualifies as a competent evaluator for specific tasks still require further standardization.”
In recent months, a series of safety incidents at AI labs—which have fueled concerns regarding the potential dangers of AI technology—has led to increasing interest in the concept of “embedded evaluators”: placing ostensibly “neutral” third-party observers directly within AI organizations. The case involving Hugging Face is one such example.
Rather than actively seeking government regulation to identify and mitigate model safety risks, these labs have instead championed third-party evaluation frameworks. However, questions persist regarding the neutrality and effectiveness of these evaluators. Anthropic’s decision to engage Accenture as its first external evaluator sparked considerable controversy, with some critics even questioning whether a consulting firm of that nature is truly qualified for the task.
Evaluation methods are shifting from testing models in isolation to assessing the entire system built around them—including frameworks and agents. Solaiman points out that frameworks (the tools users employ to access models) have a profound impact on performance; a weaker model, when paired with a superior framework, can easily outperform state-of-the-art models.
“Evaluating agents is extremely difficult and costly,” Solaiman says. “While we currently do a fair amount of model evaluation, we lack sufficient assessment of the ‘guardrails’—the safety mechanisms in place. We are working on developing better, more holistic methods for evaluating agents, as this is something that would provide significant value to many.”
In short, the current landscape of evaluation is somewhat fragmented. If benchmarks are to remain a useful tool for industry leaders, the field may need a more unified, industry-wide approach—one that ensures everyone is aligned.