Cloudflare enables content owners to charge AI crawlers for access
MAITHEM AL ANBARI (ABU DHABI)Internet company Cloudflare has launched a new 'pay-per-crawl' system, which allows website owners to charge AI crawlers whenever they seek access to their content.Pay-per-crawl will give domain owners complete control over their monetisation strategies, Cloudflare said. They can establish a consistent fee per request for their entire site. By default, this new system will prevent artificial intelligence crawlers from accessing content without the consent or financial remuneration of website owners, the company explained.Cloudflare mentioned that the request headers will indicate whether access is granted or if a payment is required, along with the associated pricing. The company will act as the Merchant of Record for the pay-per-crawl system and will also provide the necessary technical infrastructure.On average, Cloudflare handles 78 million HTTP requests per second and provides data services for 330 cities across more than 125 countries.
Approximately 16% of worldwide internet traffic is processed directly by Cloudflare's content delivery network, according to the company's estimates.
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Khaleej Times
a day ago
- Khaleej Times
How AI crawlers scrape the internet at large just changed
Cloudflare, Inc, the leading connectivity cloud company, has announced it is now the first Internet infrastructure provider to block AI crawlers accessing content without permission or compensation, by default. Starting today, website owners can choose if they want AI crawlers to access their content, and decide how AI companies can use it. AI companies can also now clearly state their purpose – if their crawlers are used for training, inference, or search – to help website owners decide which crawlers to allow. Cloudflare's new default setting is the first step toward a more sustainable future for both content creators and AI innovators. For decades, the Internet has operated on a simple exchange: search engines index content and direct users back to original websites, generating traffic and ad revenue for websites of all sizes. This cycle rewards creators that produce quality content with money and a following, while helping users discover new and relevant information. That model is now broken. AI crawlers collect content like text, articles, and images to generate answers, without sending visitors to the original source – depriving content creators of revenue, and the satisfaction of knowing someone is viewing their content. If the incentive to create original, quality content disappears, society ends up losing, and the future of the Internet is at risk. 'If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone – creators, consumers, tomorrow's AI founders, and the future of the web itself,' said Matthew Prince, co-founder and CEO of Cloudflare. 'Original content is what makes the Internet one of the greatest inventions in the last century, and it's essential that creators continue making it. AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone.' 'Cloudflare's innovative approach to block AI crawlers is a game-changer for publishers and sets a new standard for how content is respected online. When AI companies can no longer take anything they want for free, it opens the door to sustainable innovation built on permission and partnership,' said Roger Lynch, CEO of Condé Nast. 'This is a critical step toward creating a fair value exchange on the Internet that protects creators, supports quality journalism and holds AI companies accountable.' 'We have long said that AI platforms must fairly compensate publishers and creators to use our content. We can now limit access to our content to those AI partners willing to engage in fair arrangements,' said Neil Vogel, CEO of Dotdash Meredith. 'We're proud to support Cloudflare and look forward to using their tools to protect our content and the open web.' 'As the largest publisher in the country, comprised of USA TODAY and over 200 local publications throughout the USA TODAY Network, blocking unauthorized scraping and the use of our original content without fair compensation is critically important,' said Renn Turiano, Chief Consumer and Product Officer of Gannett Media. 'As our industry faces these challenges, we are optimistic the Cloudflare technology will help combat the theft of valuable IP.' 'Creators and publishers around the world leverage Pinterest to expand their businesses, reach new audiences and directly measure their success. As AI continues to reshape the digital landscape, we are committed to building a healthy Internet infrastructure where content is used for its intended purpose, so creators and publishers can thrive,' said Bill Ready, CEO of Pinterest. 'AI companies, search engines, researchers, and anyone else crawling sites have to be who they say they are. And any platform on the web should have a say in who is taking their content for what,' said Steve Huffman, co-founder and CEO of Reddit. 'The whole ecosystem of creators, platforms, web users and crawlers will be better when crawling is more transparent and controlled, and Cloudflare's efforts are a step in the right direction for everyone.' 'We applaud Cloudflare for advocating for a sustainable digital ecosystem that benefits all stakeholders — the consumers who rely on credible information, the publishers who invest in its creation, and the advertisers who support its dissemination,' said Vivek Shah, CEO of Ziff Davis. Cloudflare powers one of the world's largest networks, helping to manage and protect traffic for 20% of the web. The company handles trillions of requests daily, and thus has the world's most advanced bot management solutions, accurately distinguishing between human users and AI crawlers. In September 2024, Cloudflare introduced the option to block AI crawlers in a single click. More than one million customers have since chosen this option, meant to be an aggressive but easy solution that halts scraping while they determine their AI strategy. Now, Cloudflare is taking the next step to enforce a permission-based model for AI crawlers. AI companies will now be required to obtain explicit permission from a website before scraping. Upon sign-up with Cloudflare, every new domain will now be asked if they want to allow AI crawlers, giving customers the choice upfront to explicitly allow or deny AI crawlers access. This significant shift means that every new domain starts with the default of control, and eliminates the need for webpage owners to manually configure their settings to opt out. Customers can easily check their settings and enable crawling at any time if they want their content to be freely accessed. Now Cloudflare is making the content ecosystem more transparent for AI companies and creators. The company recently proposed new ways for AI bots to authenticate themselves as well as for websites to identify those bots – giving creators and website owners new identification mechanisms and control over what crawlers they want to allow. Cloudflare is participating in the development of a new protocol to provide bot owners and AI agent developers with a public, standard way to identify themselves.
Al Etihad
a day ago
- Al Etihad
Cloudflare enables content owners to charge AI crawlers for access
2 July 2025 17:21 MAITHEM AL ANBARI (ABU DHABI)Internet company Cloudflare has launched a new 'pay-per-crawl' system, which allows website owners to charge AI crawlers whenever they seek access to their will give domain owners complete control over their monetisation strategies, Cloudflare said. They can establish a consistent fee per request for their entire site. By default, this new system will prevent artificial intelligence crawlers from accessing content without the consent or financial remuneration of website owners, the company mentioned that the request headers will indicate whether access is granted or if a payment is required, along with the associated pricing. The company will act as the Merchant of Record for the pay-per-crawl system and will also provide the necessary technical average, Cloudflare handles 78 million HTTP requests per second and provides data services for 330 cities across more than 125 countries. Approximately 16% of worldwide internet traffic is processed directly by Cloudflare's content delivery network, according to the company's estimates.
Tahawul Tech
a day ago
- Tahawul Tech
Cloudflare introduces AI bot blocker
Millions of websites will now be able to block AI bots from accessing their content without permission thanks to a new system being rolled out by internet infrastructure firm, Cloudflare. Eventually, sites will be able to ask for payment from AI firms in return for having their content scraped. Cloudflare's tech targets AI firm bots – also known as crawlers – which are programs that explore the web, indexing and collecting data as they go. They are important to the way AI firms build, train and operate their systems. So far, Cloudflare says its tech is active on a million websites. Roger Lynch, chief executive of Condé Nast, whose print titles include GQ, Vogue, and The New Yorker, said the move was 'a game-changer' for publishers. 'This is a critical step toward creating a fair value exchange on the Internet that protects creators, supports quality journalism and holds AI companies accountable', he wrote in a statement. However, other experts say stronger legal protections will still be needed. Initially the system will apply by default to new users of Cloudflare services, plus sites that participated in an earlier effort to block crawlers. Many publishers accuse AI firms of using their content without permission. Cloudflare argues AI breaks the unwritten agreement between publishers and crawlers. AI crawlers, it argues, collect content like text, articles, and images to generate answers, without sending visitors to the original source—depriving content creators of revenue. 'If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone,' wrote the firm's chief executive Matthew Prince. To that end the company is developing a 'Pay Per Crawl' system, which would give content creators the option to request payment from AI companies for utilising their original content. According to Cloudflare there has been an explosion of AI bot activity. 'AI Crawlers generate more than 50 billion requests to the Cloudflare network every day', the company wrote in March. And there is growing concern that some AI crawlers are disregarding existing protocols for excluding bots. In an effort to counter the worst offenders Cloudflare previously developed a system where the worst miscreants would be sent to a 'Labyrinth' of web pages filled with AI generated junk. The new system attempts to use technology to protect the content of websites and to give sites the option to charge AI firms a fee to access it. Source: BBC News Image Credit: Cloudflare
