Cloudflare launches tool to help website owners monetise AI bot crawler access
has launched a tool that blocks bot crawlers from accessing content without permission or compensation to help websites make money from AI firms trying to access and train on their content, the software company said on Tuesday.
The tool allows website owners to choose whether
artificial intelligence
crawlers can access their material and set a price for access through a "pay per crawl" model, which will help them control how their work is used and compensated, Cloudflare said.
With AI crawlers increasingly collecting content without sending visitors to the original source, website owners are looking to develop additional revenue sources as search traffic referrals that once generated advertising revenue decline.
The initiative is supported by major publishers including Conde Nast and Associated Press, as well as social media companies such as Reddit and Pinterest.
Cloudflare's Chief Strategy Officer Stephanie Cohen said the goal of such tools was to give publishers control over their content, and ensure a sustainable ecosystem for online content creators and
AI companies
.
"The change in traffic patterns has been rapid, and something needed to change," Cohen said in an interview. "This is just the beginning of a new model for the internet."
Google, for example, has seen its ratio of crawls to visitors referred back to sites drop to 18:1 from 6:1 just six months ago, according to Cloudflare data, suggesting the search giant is maintaining its crawling but decreasing referrals.
The decline could be a result of users finding answers directly within Google's search results, such as AI Overviews. Still, Google's ratio is much higher than other AI companies, such as OpenAI's 1,500:1.
For decades, search engines have indexed content on the internet directing users back to websites, an approach that rewards creators for producing quality content. However, AI companies' crawlers have disrupted this model because they harvest material without sending visitors to the original source and aggregate information through chatbots such as ChatGPT, depriving creators of revenue and recognition.
Many AI companies are circumventing a common web standard used by publishers to block the scraping of their content for use in AI systems, and argue they have broken no laws in accessing content for free.
In response, some publishers, including the New York Times, have sued AI companies for
copyright infringement
, while others have struck deals to license their content.
Reddit, for example, has sued AI startup Anthropic for allegedly scraping Reddit user comments to train its AI chatbot, while inking a
content licensing
deal with Google.
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
New Indian Express
19 minutes ago
- New Indian Express
Microsoft laying off nine thousand employees
SAN FRANCISCO: Microsoft on Wednesday said it was slashing a little less than four percent of its global workforce as it seeks to cut layers of middle management and leverage new technologies. "We continue to implement organizational changes necessary to best position the company and teams for success in a dynamic marketplace," a Microsoft spokesperson said in an email. The cloud computing tech giant did not disclose the total amount of lost jobs but as of June 2024 it employed 228,000 people, bringing the latest layoffs to around 9,000 people. The job cuts follow a round in May that saw about 6,000 positions culled from its global workforce. The company, which is advancing in its plans to deploy AI across all its products, said it was working to "empower employees to spend more time focusing on meaningful work by leveraging new technologies and capabilities." "Even in the best of times, we have regularly adjusted our workforce to meet the strategic demands of the business," the company added. The company, which celebrates its 50th anniversary this year, was one of the first tech giants to double down on artificial intelligence when the launch of ChatGPT in 2022 rocked the tech industry.
Time of India
27 minutes ago
- Time of India
Zero tariffs for US: Donald Trump announces trade deal with Vietnam; latter subjected to 20% duty
US President Donald Trump announced on Wednesday that he has struck a trade deal with Vietnam. Under the terms of the deal, Vietnam will impose a 20% tariff on all goods exported to the United States which the US will have to pay zero tariffs. 'I just made a Trade Deal with Vietnam. Details to follow!' he said. Trump made the announcement through his social media platform Truth Social, declaring what he described as a major new trade agreement with the Socialist Republic of Vietnam. The deal, according to Trump, was finalised following direct talks with Vietnam's general secretary of the Communist Party, To Lam. 'It is my Great Honor to announce that I have just made a Trade Deal with the Socialist Republic of Vietnam after speaking with To Lam, the Highly Respected General Secretary of the Communist Party of Vietnam,' Trump wrote. Outlining the terms, Trump said Vietnam has agreed to pay a 20% tariff on all goods exported to the United States and a 40% tariff on any transshipped goods. The US president also claimed that Vietnam will offer the United States "TOTAL ACCESS" to its domestic markets, allowing its products to be sold there without any import tariffs. 'Vietnam will do something that they have never done before, give the United States of America TOTAL ACCESS to their Markets for Trade. by Taboola by Taboola Sponsored Links Sponsored Links Promoted Links Promoted Links You May Like Free P2,000 GCash eGift UnionBank Credit Card Apply Now Undo In other words, they will 'OPEN THEIR MARKET TO THE UNITED STATES,' meaning that, we will be able to sell our product into Vietnam at ZERO Tariff,' Trump said. He further highlighted the potential of American made SUVs in the Vietnamese market, saying that the SUV or the 'Large Engine Vehicle' that are doing well in the US, would be 'wonderful addition to the various product lines.' He ended the post with a note of appreciation, thanking To Lam. 'Dealing with General Secretary To Lam, which I did personally, was an absolute pleasure. Thank you for your attention to this matter!' The deal comes just a week ahead of the July 9 deadline, when the 90-day window for tariff negotiations was set to expire. With this agreement, Vietnam joins the UK and China as the countries to have reportedly secured a trade deal with the United States so far. Stay informed with the latest business news, updates on bank holidays and public holidays . AI Masterclass for Students. Upskill Young Ones Today!– Join Now
Indian Express
an hour ago
- Indian Express
Cloudflare will now block AI bots by default on new sites
Cloudflare, which protects around 20% of all websites globally, is changing how AI companies interact with the open web. The company will now block AI crawlers by default on all newly onboarded websites, forcing AI firms to get explicit consent before accessing site content for training or search. The move is Cloudflare's latest effort to prevent content scraping, which involves automatically extracting data from websites using AI bots without permission and usually for malicious intent. This change comes alongside the launch of a new product called Pay per Crawl, a marketplace that allows website owners and media publishers to charge AI bots for scraping their content. Publishers can set custom pricing depending on the use case – whether the data will be used to train large language models, power real-time search results, or assist independent bots. The product aims to create more transparency and fairness in how AI firms obtain data. Leading media outlets such as Condé Nast, TIME, and The Atlantic have already joined the program, blaming declining referral traffic on AI-generated answers increasingly reducing the need for users to click through to original sources. These publishers, among others, argue that without fair compensation, AI scraping undermines their ability to sustain journalism and content creation. Cloudflare also released comparative crawl-to-crawl ratios that highlight the imbalance between traffic generated and data consumed. OpenAI's crawlers reportedly make 1,700 requests per referral, while Anthropic's reach a staggering 73,000-to-1. In contrast, Google's crawl-to-click ratio sits at a more balanced 14-1. The move redefines Cloudflare's role from passive infrastructure to active negotiator in the AI data economy. As AI assistants become more independent, and as AI-generated answers increasingly replace traditional web traffic, tools like Pay per Crawl could determine which sites get fairly compensated. (This article has been curated by Arfan Jeelany, who is an intern with The Indian Express)
