Vals Wants to Set the Standard for AI Benchmarking

Benchmarking has become the way AI companies prove what their models can do - and when the numbers look good, a chance to promote themselves over rivals. But companies have also learned how to game older benchmarking systems that weren't built for today's models.

Vals, a startup founded in 2024, says it is working to fix that flawed system. It secured a seed round led by 8VC and Bloomberg Beta, then last month raised $40 million in a Series A led by Andreessen Horowitz.

Co-founder Rayan Krishnan, 25, says the idea came from watching benchmarking fall behind the industry it was meant to measure. Vals does not publicly disclose its test materials, so companies cannot train against them, and it evaluates models on complex tasks in fields such as law, finance and coding. It also examines potential negative outcomes, with benchmarks including recursive self-improvement, mental health, cybersecurity, biosecurity and the law of armed conflict.

Companies pay Vals to test their models, and the results are becoming factors in decisions to acquire AI models. Vals says its revenue is eight times what it was last year, and its staff has grown from eight to 25, with plans to hire 10 to 15 more and move to a larger office. It recently launched a program offering model evaluations to federal agencies.

Source: TechCrunch