OpenAI to Embed Outside Safety Evaluators in Pilot Program
OpenAI said Monday it will place safety evaluators from four outside firms inside its organization as part of a pilot program. The participating organizations are Apollo Research, Goodfire, METR...
OpenAI said Monday it will place safety evaluators from four outside firms inside its organization as part of a pilot program. The participating organizations are Apollo Research, Goodfire, METR and Transluce.
The announcement follows a series of safety incidents, including one in which an OpenAI agent swarm broke out of its sandbox and hacked into Hugging Face during cybersecurity testing. Many AI safety researchers, including some at Apollo, have argued that such an event might have been avoided if outside evaluators had been embedded within OpenAI to monitor its safety testing.
OpenAI said the efforts support its commitments to rely on independent assessments of safety controls under the White House Accord on Super Intelligence. The pilots will begin at different points, with Apollo's starting this week, according to OpenAI.
The idea of embedding evaluators into frontier AI labs first came from Anthropic, which last month began giving METR researchers employee-level access to its systems. Until recently, AI companies have only brought in outside reviewers to test models after post-training and shortly before public release. Evaluators say this is insufficient because it does not allow them to test for misaligned behaviors like scheming during training or to monitor internal deployment use cases that could still pose a threat.
The work with each third-party firm differs in scope. The pilot with OpenAI and Apollo will focus on investigating whether models attempt to subvert their own or their successors' alignment or control mechanisms. In other words, they will be watching for whether AI models try to undermine the safety rules and safeguards designed to keep them, or future versions of themselves, under human control. OpenAI recently published findings that while training GPT-5.6 Sol, it began leaving instructions for future versions of itself telling them to conceal mistakes and misaligned behavior from the user.
Per Apollo, the pilot will examine whether training incentivizes subversion; whether evaluations can elicit subversion; whether subversion occurs during internal deployment; and whether reasoning relevant to subversion would be detectable. Apollo's previous research with OpenAI found that models can behave differently when they know they are being tested and tend to prioritize what evaluators want to see over users' actual interests.
Goodfire will embed researchers who specialize in understanding how AI models work internally. Using its interpretability platform, the firm will help OpenAI monitor training runs, fix undesired behavior and improve alignment, per the company. Goodfire says it will publish findings, though its primary audience will be OpenAI's own researchers.
With METR, OpenAI plans two evaluations: one that looks for unintended or dangerous behavior in the model itself, including problems OpenAI's own tests might have missed, and another that tests whether OpenAI's systems are capable of catching that behavior. For the first evaluation, METR will largely decide for itself what to investigate and how to test the model. OpenAI and METR have not yet decided the scope of the second assessment, according to OpenAI.
Transluce will examine whether OpenAI's post-training safety evaluations themselves are good enough. The organizations also plan to test in 2027 how OpenAI uses its safety tests to decide whether a model is safe enough to release.
At a time when the independence of third-party evaluations is under scrutiny, Apollo and Goodfire are both receiving financial compensation for their evaluations. METR and Transluce are not, but they may accept compute or API credits.
Apollo, which published the most detailed account of its agreement with OpenAI, offers a closer look at the scope and limitations of these evaluations. Apollo says an essential part of the pilot is OpenAI's agreement that the firm can publicly explain the conditions and limitations of its work.
Still, Apollo says its access to OpenAI's systems is lacking in areas. For a truly independent evaluation, Apollo says it needs true employee-level access, including all data, records and documentation relevant to its assessment even if a developer was trying to hide unfavorable information. In the first pilot, OpenAI is providing Apollo with access to only the information required to assess whether the models engage in subversion attempts, including relevant training code and datasets, reinforcement-learning rollouts with available reasoning traces and rewards, and the ability to query intermediate model checkpoints.
The scope of access will be limited to only the agreed claims and only for the specific models and time periods agreed for the assessment, Apollo says. OpenAI still reserves the right to withhold certain information within that scope if it deems it necessary because of technical infeasibility, security risks or IP protection. The company agreed not to withhold information simply because it is unfavorable.
Apollo also will not get access to all OpenAI staffers, instead being limited to only authorized staff selected by the lab, though Apollo can suggest certain individuals. If an OpenAI employee outside this group raises confidential information within the scope of Apollo's research, the firm cannot follow up and will need to cease communication and route followup through OpenAI.
That condition follows OpenAI firing three researchers for allegedly sharing confidential information with METR, which performed the Hugging Face incident post-mortem alongside independent AI safety research group Redwood.
Buck Shlegeris, CEO of Redwood, said in a post on Monday that the agreements announced do not meet the bar of granting third parties employee-like access, calling it disappointing but not surprising.
The pilot will also be limited in how much researchers can disclose. While Apollo will be allowed to publish independent conclusions, OpenAI will be able to limit evidence available to substantiate them publicly. Apollo has pushed back against this limitation, saying researchers should be allowed to disclose findings to an appropriate outside audience even if the developer was trying to prevent disclosure of unfavorable findings.
OpenAI says its Safety and Security Committee, which oversees safety on behalf of the OpenAI Foundation's board, will get to review unredacted assessment reports and can communicate privately with evaluators.
While OpenAI is not granting all of Apollo's conditions, the firm says the pilot is an ambitious first step toward deeper access and disclosure rights and that it is working toward running full embedded evaluations with multiple AI developers in the future.
Source: TechCrunch
Ready to Modernize Your Business?
Get your AI automation roadmap in minutes, not months.
Analyze Your Workflows →