Embedded AI Evaluators Face Trust and Standardization Hurdles, Hugging Face Policy Chief Says

AI companies have built their reputations on model benchmarking, the evaluation systems meant to neutrally and accurately measure whether one AI product outperforms another. But the methods behind those evaluations remain complicated and still maturing, said Irene Solaiman, Hugging Face's chief policy officer, during a discussion at TechCrunch Disrupt last week.

Asked whether current tests can be trusted, Solaiman called trust "such a weighted term" and pointed to reproducibility as a major factor - whether an evaluation's results can be reproduced and whether there is enough access to the model itself, which she said is where open weights matter greatly. Reproducibility has long been one of the stickier aspects of benchmarking and remains heavily debated in the industry.

Beyond any single test's validity, Solaiman said evaluations are "an evolving science" that no one yet knows how to meaningfully compare to each other. She said they "start to get ineffective relatively quickly" and that the field lacks as thriving an ecosystem as it needs. Demand for an embedded evaluator ecosystem is growing, she added, but the qualifications that determine who counts as an effective evaluator across tasks need far more standardization.

Embedded evaluators - supposedly neutral third-party observers placed inside AI organizations - have gained popularity over recent months as labs confront a run of safety incidents. Hugging Face itself was famously the target of one such incident. Rather than active government regulation, labs have pushed the third-party approach, though doubts persist about how neutral or effective those evaluators will be. Anthropic's choice of Accenture as its first evaluator drew backlash, with critics mocking the consulting firm's qualifications for the task.

Evaluations are also shifting from testing the model alone to testing the entire system around it, including the harness and agent. Solaiman noted that the harness - the tool a user employs to access the model - heavily influences performance, and a weaker model with a better harness could easily outperform a frontier model. She said agentic evaluations are "really hard and really expensive," with far more evaluations for models than for harnesses, while better evaluations for the holistic agent are being developed because that is where many people are getting value.

In short, the evaluation space is something of a free-for-all at the moment, and if benchmarking is to remain a useful tool for industry leaders, a more uniform, industry-wide approach may be necessary.

Source: TechCrunch