Cloudflare launches tool to help website owners monetize AI bot crawler access
NEW YORK (Reuters) -Cloudflare has launched a tool that blocks bot crawlers from accessing content without permission or compensation to help websites make money from AI firms trying to access and train on their content, the software company said on Tuesday.
The tool allows website owners to choose whether artificial intelligence crawlers can access their material and set a price for access through a "pay per crawl" model, which will help them control how their work is used and compensated, Cloudflare said.
With AI crawlers increasingly collecting content without sending visitors to the original source, website owners are looking to develop additional revenue sources as search traffic referrals that once generated advertising revenue decline.
The initiative is supported by major publishers including Condé Nast and Associated Press, as well as social media companies such as Reddit and Pinterest.
Cloudflare's Chief Strategy Officer Stephanie Cohen said the goal of such tools was to give publishers control over their content, and ensure a sustainable ecosystem for online content creators and AI companies.
"The change in traffic patterns has been rapid, and something needed to change," Cohen said in an interview. "This is just the beginning of a new model for the internet."
Google, for example, has seen its ratio of crawls to visitors referred back to sites drop to 18:1 from 6:1 just six months ago, according to Cloudflare data, suggesting the search giant is maintaining its crawling but decreasing referrals.
The decline could be a result of users finding answers directly within Google's search results, such as AI Overviews. Still, Google's ratio is much higher than other AI companies, such as OpenAI's 1,500:1.
For decades, search engines have indexed content on the internet directing users back to websites, an approach that rewards creators for producing quality content. However, AI companies' crawlers have disrupted this model because they harvest material without sending visitors to the original source and aggregate information through chatbots such as ChatGPT, depriving creators of revenue and recognition.
Many AI companies are circumventing a common web standard used by publishers to block the scraping of their content for use in AI systems, and argue they have broken no laws in accessing content for free.
In response, some publishers, including the New York Times, have sued AI companies for copyright infringement, while others have struck deals to license their content.
Reddit, for example, has sued AI startup Anthropic for allegedly scraping Reddit user comments to train its AI chatbot, while inking a content licensing deal with Google.
(Reporting by Krystal Hu in New York; Editing by Kate Mayberry)
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Free Malaysia Today
an hour ago
- Free Malaysia Today
Alibaba expands AI cloud services in Malaysia, Philippines
Alibaba's cloud unit has launched its third data centre in Malaysia this week. (EyePress pic) BEIJING : Alibaba Group Holding Ltd is adding new data centres in Malaysia and the Philippines in pursuit of artificial intelligence (AI)-driven growth. 'The Hangzhou-based company's cloud unit launched its third data centre in Malaysia this week and it also plans to open its second data centre in the Philippines in October,' it said in a statement released today. Alibaba Cloud also said it's launching a global competency centre in Singapore to help accelerate AI adoption across industries. It said the centre would help more than 5,000 businesses and 100,000 developers access advanced AI models. Best known for its e-commerce operations in China, Alibaba has been charging into AI, building standalone offerings around its Qwen AI models and growing its cloud services. It has also announced infrastructure investments in Thailand, Mexico and South Korea. In the wake of DeepSeek's emergence on the global stage, Alibaba CEO Eddie Wu declared in February the company's 'primary objective' is now artificial general intelligence, a goal in the industry to build AI systems with human-level intellectual capabilities.
Malay Mail
2 hours ago
- Malay Mail
Jury finds Google liable for covert data collection from idle Android phones, awards US$314m to California users
SAN FRANCISCO, July 2 — A jury in San Jose, California, said yesterday that Google misused customers' cell phone data and must pay more than US$314.6 million (RM1.5 billion) to Android smartphone users in the state, according to an attorney for the plaintiffs. The jury agreed with the plaintiffs that Alphabet's Google was liable for sending and receiving information from the devices without permission while they were idle, causing what the lawsuit had called 'mandatory and unavoidable burdens shouldered by Android device users for Google's benefit.' Google spokesperson Jose Castaneda said in a statement that the company would appeal, and that the verdict 'misunderstands services that are critical to the security, performance, and reliability of Android devices.' The plaintiffs' attorney Glen Summers said the verdict 'forcefully vindicates the merits of this case and reflects the seriousness of Google's misconduct.' The plaintiffs filed the class action in state court in 2019 on behalf of an estimated 14 million Californians. They argued that Google collected information from idle phones running its Android operating system for company uses like targeted advertising, consuming Android users' cellular data at their expense. Google told the court that no Android users were harmed by the data transfers and that users consented to them in the company's terms of service and privacy policies. Another group filed a separate lawsuit in federal court in San Jose, bringing the same claims against Google on behalf of Android users in the other 49 states. That case is scheduled for trial in April 2026. — Reuters
The Star
2 hours ago
- The Star
Google ends recipe pilot that left creators fearing web-traffic hit
The company's retreat on the recipe feature comes amid a larger debate about whether the terms of engagement between the search giant and publishers should be renegotiated as generative AI remakes the web. — Pixabay Google has ended tests of a feature that would have let users open a snapshot of cooking-recipe content directly in web search results – a welcome development for creators and food bloggers who were concerned about eroding traffic to their sites. In recent months, Alphabet Inc-owned Google has tested Recipe Quick View, which showed some food bloggers' content in search. The company framed the feature as an attempt to help users determine whether they are interested in a recipe before visiting a website. But some bloggers said they feared that the product would keep users from clicking through to their sites, depriving them of traffic and ad revenue. Google on Tuesday confirmed it ended the trial. "We continually experiment with ways to make it easier for people to find helpful information on Search,' a spokesperson for Google said in a statement. "Learnings from these experiments help to inform future development and efforts.' The company's retreat on the recipe feature comes amid a larger debate about whether the terms of engagement between the search giant and publishers should be renegotiated as generative AI remakes the web. Creators who depend on Google and other technology companies for traffic have spoken out about the devastating toll AI has taken on their businesses, and are pushing for more control over how their content is used to develop the technology. Cloudflare Inc, a web infrastructure and security company, on Tuesday announced a "pay per crawl' program that allows creators to bill AI services for access to their content. "We have this ongoing standards conversation of how might a site say, 'I want you to do search but not generative AI training,'' said Derek Slater, co-founder of Proteus Strategies, a tech policy consulting firm. "Those are moving technological and market-driven pieces that have not settled yet.' Creators who participated in the pilot with Google were being directly compensated, but not nearly enough to offset declines in their advertising revenue, Marc McCollum, chief growth officer at Raptive, told Bloomberg earlier this year. Raptive, a media company that represents creators, estimated that traffic to food blogs would fall by half if the feature were implemented in full. Soon after news of the test program surfaced, creators made their concerns known, said Tomiko Harvey, a travel and food blogger at Passports and Grub, who attended a Google creator summit in May. "I'm actually feeling hopeful about this decision from Google,' Harvey wrote in an email. "Canceling the Quick View Recipe feature feels like a rare but welcome acknowledgment that creator feedback matters.' Although Google's decision has some creators feeling relief, they likely haven't seen the last of the search giant's experimentation in a vibrant corner of the web. In recent years, many food bloggers have padded their recipes with lengthy preambles to ensure their content ranks well in Google Search, irritating readers in the process. As it crawls the web, Google will glean information about recipes' core principles – say, the basic proportions of broth and meat in a recipe for chicken noodle soup – which it can then use to inform its own answers, Slater said. "Recipes are, by definition, facts, and facts are not copyrightable,' Slater said. For Google, "there's always going to be a tension between giving the user exactly what they want,' he added, "and directing the user to another site.' – Bloomberg
