As artificial intelligence companies compete to build increasingly powerful models, benchmark scores have become a major way to demonstrate progress. But according to Hugging Face’s chief policy officer, Irene Solaiman, the industry still has a long way to go before these evaluations can be considered consistently reliable.
Solaiman warned that AI benchmarking remains an evolving science, with unresolved questions about reproducibility, fairness, and whether different tests can meaningfully be compared.
Her concerns also extend to a growing industry proposal: embedding independent evaluators inside AI companies to monitor model safety and performance.
Can AI Benchmark Results Be Trusted?
AI benchmarks are designed to measure how well models perform on tasks such as reasoning, coding, and problem-solving. Companies often use the results to compare their products with competing systems.
However, a high score doesn’t necessarily mean a model will perform equally well in real-world situations.
Solaiman said one important factor in determining whether an evaluation can be trusted is reproducibility—whether other researchers can repeat the test and obtain comparable results.
Access to a model also matters. Open-weight systems can give researchers greater opportunities to inspect and test model behavior independently, although access alone doesn’t guarantee that an evaluation is reliable.
Another challenge is the lack of consistent methods for comparing different benchmarks. Tests may use different datasets, procedures, or evaluation criteria, making direct comparisons difficult.
Why Embedded AI Evaluators Face Challenges
Some AI companies are exploring the idea of placing third-party evaluators within their organizations to provide independent oversight of model safety.
The approach has gained attention as concerns grow about AI-related security incidents and the potential risks of increasingly capable systems.
However, Solaiman cautioned that simply introducing evaluators into AI companies won’t automatically solve the problem.
The industry still needs clearer standards for determining who qualifies as an effective evaluator, which tasks they should assess, and how their independence and performance should be measured.
The appointment of consulting firm Accenture as Anthropic’s first embedded evaluator also attracted criticism from people questioning whether the arrangement would deliver meaningful independent oversight.
The broader concern is whether evaluators working inside the organizations they assess can remain sufficiently independent and provide credible findings.
Testing the Entire AI System, Not Just the Model
Another major challenge is that AI performance increasingly depends on more than the underlying model.
Modern AI products often combine a model with a harness—the software and tools used to interact with it—as well as agents that can execute tasks, use external services, and perform multiple steps autonomously.
These components can substantially influence the final result.
A less powerful model operating within a well-designed system could outperform a more advanced model paired with weaker tools or an inefficient workflow.
That means evaluating the model alone may not accurately reflect the performance users experience.
Solaiman noted that agent-based evaluations are particularly difficult and expensive. While the industry has developed many ways to test individual models, evaluations of the surrounding software systems and complete AI agents remain less mature.
The Need for Industry-Wide Standards
The growing complexity of AI systems makes consistent evaluation methods increasingly important.
Without shared standards, companies may rely on benchmarks that favor their own models, produce results that are difficult to reproduce, or fail to measure capabilities that matter in practical use.
A stronger evaluation framework would need to address several areas:
- Reproducibility: Researchers should be able to repeat tests and verify reported results.
- Comparability: Benchmarks should use sufficiently consistent methods to support meaningful comparisons.
- Independent oversight: Evaluators should have clear standards and safeguards for maintaining credibility.
- Real-world performance: Testing should cover the complete AI system, including tools, harnesses, and autonomous agents.
These measures would not eliminate every limitation, but they could make evaluations more useful for researchers, businesses, policymakers, and consumers.
AI Benchmarking Needs More Than Bigger Scores
AI benchmarks will likely remain an important part of measuring technological progress. However, increasingly sophisticated systems require more comprehensive testing methods.
Solaiman’s warning highlights a central issue facing the industry: measuring AI performance is not the same as proving that an AI system is reliable or safe.
Until evaluation methods become more consistent and transparent, benchmark scores and embedded oversight programs will need to be interpreted carefully.
The next stage of AI evaluation may depend less on creating more tests and more on developing standards that make their results reproducible, comparable, and meaningful in real-world conditions.
