OpenAI Makes Internal AI Progress Metrics Public in Push for Shared Safety Standards
Modern AI systems may look like a patchwork of specialised techniques, but OpenAI argues they largely draw on one simpler process: training a compressor on enormous amounts of data. In a blog...
Modern AI systems may look like a patchwork of specialised techniques, but OpenAI argues they largely draw on one simpler process: training a compressor on enormous amounts of data. In a blog post, the company makes the case for perplexity - also known as cross-entropy loss - as a shared yardstick for comparing model developers and informing common safety standards.
Perplexity measures how well a model predicts the next token; lower scores mean better predictions. OpenAI says it weighed the competitive risk of publishing its internal numbers against the urgency of industry-wide safety measures, and decided transparency won out.
The company traces the metric across its models: GPT-3.5 reached a log-perplexity of about 0.99 on its main evaluation dataset and powered the original ChatGPT. GPT-4 came in at roughly 0.91 and showed strong chain-of-thought reasoning. Losses below 0.8 have coincided with striking capabilities in cybersecurity and mathematics. OpenAI stresses it was not obvious beforehand that these abilities would appear at those thresholds.
OpenAI also flags a possible acceleration from AI contributing to pretraining research. It says a model at about 0.76 log-perplexity began producing groundbreaking mathematics results, and suggests similar leaps in deep learning research could conceivably arrive as soon as 2027. The company expects further transformative capabilities in the ~0.68-0.7 window, while warning that alignment auditing may degrade. It proposes shared transparency norms and common development limits tied to perplexity until monitoring capabilities reach a concrete milestone.
The post acknowledges challenges: numbers are hard to compare across labs, perplexity does not map directly to specific tasks, it misses gains from chain-of-thought reasoning, and the data's inherent entropy floor is unknown. Still, OpenAI believes the metric has stood the test of time and can help standardise how the industry discusses AI progress.
Source: OpenAI Blog
Ready to Modernize Your Business?
Get your AI automation roadmap in minutes, not months.
Analyze Your Workflows →