Cloudflare experiment will block AI bot scrapers unless they pay a fee

Engadget

a day ago

Cloudflare has rolled out a couple of new measures meant to keep AI bot crawlers at bay. To start with, every new domain customer that signs up with the company to manage their website traffic will now be asked if they want to allow AI crawlers or to block them altogether. The company released a free tool in 2024 to block AI bots, but with this change, users can block them by default without having to tinker with their settings. Several big publishers, including Condé Nast, TIME and The Associated Press have already signed up to block crawlers. In addition, Cloudflare has launched a private beta experiment called "pay per crawl," which would only allow crawlers to access a website's content if they pay for it.
Matthew Prince, Cloudflare's CEO, recently went on record to say that publishers are facing an existential threat, because people aren't clicking on chatbots' source links. If users don't visit those sources, the websites don't get the ad revenue they need to be able to keep running. "Original content is what makes the Internet one of the greatest inventions in the last century, and it's essential that creators continue making it," Prince said in a statement released with the company's latest updates. "AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone."
Cloudflare believes publishers should be able to charge AI bots for access if they want to, and pay per crawl is its first experiment for that particular purpose. "Each time an AI crawler requests content, they either present payment intent via request headers for successful access (HTTP response code 200), or receive a 402 Payment Required response with pricing," Cloudflare explained. The company records those transactions and provides the underlying technical infrastructure. Publishers will be able to allow certain crawlers to access their content for free if they want to, and they can define a flat, per-request price across its websites for other crawlers.
The company says pay per crawl is still in its very early stages, and it expects the tool to evolve in the future. It also says that it supports the development of other marketplaces and ways to charge AI crawlers for content. A marketplace could, for instance, allow dynamic pricing that enable publishers to charge different rates for different types of content.