Anthropic Proposes Open Source Tool to Verify Which AI Model Made an Output

Anthropic is developing an open source system that would let users confirm which AI model actually produced a given output, the company said in a blog post published Tuesday.

The system, called FlashProve, is designed to deliver what researchers describe as "provable inference" - a cryptographic, mathematical guarantee that an output came from a specific model. The idea is similar to cryptographic email signatures that confirm a message came from its claimed sender, and it would resist impostors trying to pass one model off as another, even if the impostor is the lab running the model.

Researchers say the system could stop an AI company from quietly swapping in a cheaper model to save money, or a pricier one to make its performance look better. It could also verify third-party claims - for example, a developer saying a frontier cybersecurity model scanned their software library for vulnerabilities, or a company saying an appropriate model approved its business plan.

For the system to work, AI labs would publish a "fingerprint" of each model they release, which could later be matched to outputs and checked cryptographically. The fingerprints do not expose a model's weights - the internal numbers that shape its behavior and that closed-model makers like Anthropic and OpenAI keep private - so labs could publish them without giving up their most valuable asset, allowing the system to work with commercial models too. It is also built to be lightweight enough for verification on a local device.

Anthropic released a detailed technical specification and working code for parts of the system under an open source license on GitHub. Research is ongoing, and there are no plans to deploy it at Anthropic or any other lab. "This research post is meant to share intermediate progress, and not a completed research agenda," researchers wrote, adding that they are sharing it so others can take the work and improve it.

Source: TechCrunch