Google, OpenAI, Anthropic and Microsoft Back Shared AI Scraping Signal

Google, Anthropic, OpenAI and Microsoft have all said they will honor a new robots.txt directive that lets publishers state whether their content may be scraped for generative AI training. The companies are adopting a proposal under consideration at IETF, the internet's open standards body, written by Anthropic technical lead for standards Kevin Kelley.

Today, site owners who want to opt out must identify each provider's crawler and write separate rules for every one. Kelley's proposal replaces that patchwork with a single signal that multiple AI companies recognize. Publishers add a new user agent to their robots.txt file, dedicated to AI training, and can then disallow or allow crawling for that purpose, including for specific sections such as a press kit.

The directive blocks crawlers gathering data for AI training only. It does not stop crawling altogether or keep content out of Google search. Google said it will read the signal as a stop sign aimed specifically at its Google-Extended user agent, which collects material used to train Gemini models.

The signal also cannot undo scraping that has already happened: content the companies have already crawled stays with them. It only governs future scraping of sites or pages. Anthropic has already begun supporting the directive. Microsoft said it will start on February 1, 2027, and Google said it will follow in early 2027, while OpenAI pledged to implement the signal in the coming months.

Source: Engadget