AI companies have built their credibility on model benchmarking — the evaluation systems meant to determine whether one AI product outperforms another — but these methods remain complicated and immature, according to Irene Solaiman, Hugging Face's chief policy officer, speaking at TechCrunch Disrupt last week. Testing protocols need to evolve into a more consistent discipline if the sector plans to depend on them going forward, Solaiman said.
The challenges start with reproducibility, which has remained one of the tougher problems in benchmarking and continues to spark heated discussion across the industry. Beyond the validity of any single test, there's the question of how different evaluations stack up against one another. "They are an evolving science — where we don't know how to meaningfully compare them to each other," Solaiman said. Tests lose their effectiveness fairly quickly, and the industry lacks the robust ecosystem it requires, she added. The push for embedded evaluators — supposedly neutral third-party observers stationed inside AI organizations — has gained momentum in recent months as labs face a wave of safety incidents that have raised alarms about potential harms from their technology. Yet questions persist about how impartial or capable these evaluators will actually prove to be. Anthropic's choice to bring on Accenture as its first evaluator triggered considerable criticism, with skeptics questioning whether the consulting company possessed the necessary expertise.
According to Solaiman, trust in evaluation results hinges partly on whether those results can be replicated and whether researchers have sufficient access to the model itself — which is where open weights play a major role. Evaluations are now shifting from simply testing the model to assessing the complete system surrounding it, including the harness and agent. The harness — the tool users employ to interact with the model — significantly influences performance, and an inferior model paired with a superior harness could easily beat a cutting-edge model, Solaiman explained. "Agentic evals are really hard and really expensive," she said, noting that the industry has more evaluations for models but fewer for harnesses, while better assessments for the complete agent are still in development because that's where many people derive value.
The evaluation landscape currently operates as something of a free-for-all, and if benchmarking is going to maintain its status as a valuable resource for industry leaders, a more standardized, sector-wide approach where all participants share common ground may be essential. The hunger for an embedded evaluator ecosystem exists now, but the criteria that define who qualifies as an effective evaluator across various tasks require substantially more standardization, according to the Hugging Face policy chief. Without a unified framework, the industry risks relying on measurement systems that can't deliver the consistency and comparability that meaningful progress demands. Companies betting their reputations on benchmark scores may find themselves navigating a landscape where the rules of measurement remain frustratingly unclear and the path to genuine accountability stays elusive.