Researchers Uncover Method to Expose Hidden Reasoning in AI Models

Computer scientists have found a way to extract the concealed "thinking" that advanced AI models use when solving complex problems, according to a recent study. The technique offers some evidence - though not definitive proof - that certain Chinese models may have been trained by "distilling" reasoning information from US models, as their thought patterns appear closely aligned.

The researchers also demonstrated that the method could retrieve personal data like passwords and API keys from a model's internal reasoning, though this particular vulnerability has since been patched. Alexander Panfilov, a computer scientist at the University of Tübingen in Germany, noted that all major frontier model providers tested shared this issue, which could lead to information leakage and enable large-scale reasoning distillation attacks.

The team, which included experts from the University of Tübingen, the Max Planck Institute, MATS Research, and security firm Snyk, identified the problem in models from OpenAI, Anthropic, and Google accessed via APIs. In their paper, they showed that the open-weight Chinese model Kimi K3 from Moonshot AI produced outputs strikingly similar to the hidden reasoning traces of Claude Opus 4.8 and GPT 5.6 Sol for certain prompts, though they stressed this "cannot causally establish distillation." Two other open-weight models - China's DeepSeek and US-based Thinking Machines' Inkling - did not show such similarities. Moonshot AI and Z.ai did not respond to requests for comment.

Distillation is a common technique for copying model capabilities, but it has become controversial amid claims that Chinese firms use it to replicate top US models. OpenAI told lawmakers in February that DeepSeek may have copied its model for R1, and Anthropic made similar allegations about Alibaba's Qwen in June. There is no indication the specific technique was used, but the researchers say it could enable more extensive distillation from closed models than previously thought.

The attack exploits the fact that companies offer smaller, less-aligned versions of their models. By feeding encrypted reasoning traces to these weaker variants, the researchers could reveal the hidden thoughts. The same method exposed API keys and passwords embedded in reasoning traces. The team alerted the companies last month, and each has adjusted its API to mitigate the issue, though some reasoning traces can still be uncovered. Anthropic's spokesperson Michael Aciman said the company values independent research and is building short-term mitigations, adding that the study did not involve recovering encryption keys or personal data from its systems. Google and OpenAI declined to comment.

Distillation has become a geopolitical flashpoint as US and Chinese firms compete for AI dominance. China hawks argue it gives the country an edge, while others, like Meta CEO Mark Zuckerberg, say it's essential to the open-source ecosystem. Kyle Miller of CSET noted uncertainty about how much distillation actually benefits Chinese labs, suggesting removing it wouldn't drastically change the competitive landscape. The researchers tested 90 questions per model, finding Kimi K3 often generated answers remarkably close to the proprietary reasoning traces. Oxford's Yarin Gal added that blocking distillation could slow AI progress overall. Despite potential restrictions, the study suggests AI models may continue to expose their inner thinking in unexpected ways.

Source: Wired