Study Finds AI Watermarking Can Weaken Safety Guardrails

New research suggests that watermarking AI-generated text can alter how models respond to harmful prompts, including making them more likely to comply with requests they would normally refuse.

The findings come as platforms adopt watermarking to comply with a new European Union law. Anthropic has said future Claude models will use SynthID-Text, an open-source approach from Google that uses a secret key to subtly influence next-word selection so outputs can be identified.

Andrea Siposova, an AI security researcher at Lasso Security, tested SynthID-Text's "non-distortionary" configuration through Hugging Face's unmodified SynthIDTextWatermarkLogitsProcessor. She fed harmful prompts into six open-weight models and compared responses with and without watermarking. The effect was more pronounced when prompts used injection techniques, with several models becoming more likely to answer harmful requests they would otherwise refuse. Responses also varied depending on which secret key was used.

The researcher calls this behavioral effect "sampling drift," warning it can affect not only what a model says but which tools an agent calls and what arguments it passes. The study did not test Claude models or the specific SynthID implementation they will use, and it relied on Hugging Face's implementation. Still, the results indicate that red-team exercises should stress-test platforms with watermarking enabled.

Source: Ars Technica