Cloudflare launches a marketplace that lets websites charge AI bots for scraping
For the last year, Cloudflare has launched tools for publishers to address the rampant rise of AI crawlers, including a one-click solution to block all AI bots, as well as a dashboard to view how AI crawlers are visiting their site. In a 2024 interview, Cloudflare CEO Matthew Prince told TechCrunch these products were laying a foundation for a new type of marketplace in which publishers could distribute their content to AI companies and be compensated for it.
Now, Cloudflare is bringing that marketplace to life.
It's called Pay per Crawl, and Cloudflare is launching the 'experiment' in private beta on Tuesday. Website owners in the experiment can choose to let AI crawlers, on an individual basis, scrape their site at a set rate — a micropayment for every single 'crawl.' Alternatively, website owners can choose to let AI crawlers scrape their site for free, or block them altogether. Cloudflare claims its tools will let website owners see whether crawlers are scraping their site for AI training data, to appear in AI search responses, or for other purposes.
At scale, Cloudflare's marketplace is a big idea that could offer publishers a potential business model for the AI era — and it also places Cloudflare at the center of it all. The launch of the marketplace comes at a time when news publishers are facing existential questions about how to reach readers, as Google Search traffic fades away and AI chatbots rise in popularity.
There's not a clear answer for how news publishers will survive in the AI era. Some, such as the New York Times, have filed lawsuits against tech companies for training their AI models on news articles without permission. Meanwhile, other publishers have struck multi-year deals to license their content for AI model training and to have their content appear in AI chatbot responses.
Even so, only large publishers have struck AI licensing deals, and it's still unclear whether they provide meaningful sources of revenue. Cloudflare aims to create a more durable system where publishers can set prices on their own terms.
The company also announced Tuesday that new websites set up with Cloudflare will now, by default, block all AI crawlers. Site owners will have to grant certain AI crawlers permission to access their site — a change Cloudflare says will give every new domain 'the default of control.'
Several large publishers, including Conde Nast, TIME, The Associated Press, The Atlantic, ADWEEK, and Fortune, have signed on with Cloudflare to block AI crawlers by default in support of the company's broader goal of a 'permission-based approach to crawling.'
The business model that many of these publishers relied on for decades is slowly becoming unreliable. Historically, online publishers have allowed Google to scrape their sites in return for referrals in Google Search, which translated to traffic to their sites, and ultimately, ad revenue.
However, new data from Cloudflare suggests that publishers may be getting a worse deal in the AI era than in the Google Search era. While some websites cite ChatGPT as a major traffic source, that doesn't appear to be the case broadly.
This June, Cloudflare says it found that Google's crawler scraped its websites 14 times for every referral it gave them. Meanwhile, OpenAI's crawler scraped websites 17,000 times for every one referral, while Anthropic scraped websites 73,000 times for every referral.
Meanwhile, OpenAI and Google are building AI agents that are designed to visit websites on behalf of users, collect information, and deliver it back to users directly. A future in which these tools are mainstream has huge implications for publishers that rely on readers visiting their sites.
Cloudflare notes that the 'true potential' of Pay per Crawl may emerge in an 'agentic' future.
'What if an agentic paywall could operate at the network edge, entirely programmatically? Imagine asking your favorite deep research program to help you synthesize the latest cancer research or a legal brief, or just help you find the best restaurant in Soho — and then giving that agent a budget to spend to acquire the best and most relevant content,' Cloudflare said in a blog post.
To participate in Cloudflare's experimental marketplace, AI companies and publishers must both be set up with Cloudflare accounts. In their accounts, both parties can set rates at which they'd like to buy and sell a 'crawl' of the publisher's content. Cloudflare acts as the intermediary in these transactions, charging the AI company and distributing the earnings to the publisher.
Cloudflare spokesperson Ripley Park tells TechCrunch there are no stablecoins or cryptocurrency involved in Pay per Crawl at this time, even though many have suggested digital currency would be perfect for something like this.
Cloudflare's marketplace feels like a bold vision for the future that requires a lot of publishers and AI companies to get on board. Still, there's no guarantee publishers will get a good deal, and convincing AI firms to participate could be tough, given they're currently scraping content for free.
Nevertheless, Cloudflare seems like one of the few companies in a position to make a marketplace like this happen.
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Business Insider
31 minutes ago
- Business Insider
OpenAI employees share their 3 favorite tips for using ChatGPT
If you ever happen to see Nick Turley, the head of ChatGPT at OpenAI, muttering to himself on a weekday morning, it might be because he's talking to a chatbot. Turley said that ChatGPT's voice feature is his favorite tip for using the technology on a recent episode of the OpenAI podcast. "On my way to work, I'll use it to process my own thoughts. With some luck, and I think this works most days, I'll have the restructured list of to-dos by the time I actually get there," he said, adding that the voice feature isn't yet mainstream because there are a bunch of small "kinks" still. He said he finds it valuable to force himself to articulate his thoughts aloud, and wants to see the feature improve next year. Mark Chen, OpenAI's chief research officer, said on the podcast that he's a fan of the deep research feature, especially before an introduction. "When I go meet someone new, when I'm going to talk to someone about AI, I just preflight topics," Chen said. "I think the model can do a really good job of contextualizing who I am, who I'm about to meet, and what things we might find interesting." And podcast host Andrew Mayne, who was formerly OpenAI's science communicator and worked on ChatGPT, said he uses the technology when he's out at a restaurant. "I take a photograph of a menu and I'm like, 'Help me plan a meal or whatever, I'm trying to stick to a diet," Mayne said. Turley, however, cautioned against using the same trick for the wine list. "It keeps embarrassing me with hallucinated wine recommendations, and I go order it and they're like, 'Never heard of this one,'" he said. Corporate executives across companies are using AI in their daily lives, and OpenAI CEO Sam Altman is no different. Altman said on the "ReThinking" podcast in January that he uses it in "the boring ways," for things like processing emails and summarizing documents. When Altman spoke on the OpenAI podcast in June, he said that he uses ChatGPT "constantly" as a father. At the time, he said he was mainly using it to research developmental stages. "Clearly, people have been able to take care of babies without ChatGPT for a long time," Altman said. "I don't know how I would have done that."
Yahoo
31 minutes ago
- Yahoo
McDonald's is adding a Spicy McMuffin to its breakfast lineup
You can find original article here Nrn. Subscribe to our free daily Nrn newsletter. McDonald's 50th anniversary celebration of its signature Egg McMuffin continues with the introduction of a Spicy McMuffin, available starting July 8 at participating restaurants for a limited time. The offering includes melted cheese, an egg, Canadian bacon, and Spicy Pepper Sauce served on a toasted and buttered English Muffin. The summer lineup also includes the Spicy Sausage McMuffin and the Spicy Sausage McMuffin with Egg. According to the company, the popular Egg McMuffin was created by Herb Peterson, a franchisee in Santa Barbara, Calif., who presented the concept to founder Ray Croc in 1971. The idea was to create a portable Eggs Benedict. Breakfast is a priority for McDonald's value platform, as its McValue menu, introduced in January, includes items like the Sausage McMuffin, Sausage Biscuit, Sausage Burrito, and Hash Browns, available for a "Buy One, Add One for $1" deal. During the company's fourth quarter earnings call earlier this year, chief financial officer Ian Borden said breakfast has been a strong daypart for the domestic business. 'It's an area where we're taking share,' he said. During the first quarter earnings call in late April, however, chief executive officer Chris Kempczinski said there has been a decline in frequency during the 'bellwether' breakfast daypart. 'Morning now is a place that you're seeing people are choosing either to skip breakfast or they're choosing to eat at home for breakfast. And I think that's more to explain what's going on in the U.S. versus any kind of segment shift,' he said. The Spicy McMuffin comes on the heels of two other menu launches for McDonald's, a new Blueberry & Crème Pie and a new premium burger called the Daily Double. These launches come as the chain has experienced two rare and consecutive quarters of same-store sales declines and traffic softness. Contact Alicia Kelso at Sign in to access your portfolio
Yahoo
31 minutes ago
- Yahoo
Bitcoin Bulls Should Be Wary of the Dollar Index's Death Cross: Technical Analysis
Bitcoin (BTC) bulls are optimistic that the U.S. dollar's broader sell-off will continue in the second half of the year, fueling the crypto bull run. However, one chart warrants caution when relying on bearish dollar predictions. That's the dollar index's weekly chart, which shows the 50-week simple moving average (SMA) is on track to cross below the 200-week SMA in the near term, forming the notorious death cross. While the ominous-sounding pattern is widely regarded as a long-term bearish signal, historically, it has proven to be a bear trap, consistently marking bottoms and bullish trend reversals in the US dollar. The chart below shows that DXY has chalked out four weekly chart death crosses since 2009, and each of those marked the end of downtrends (marked by vertical lines), setting the stage for sharp rallies. The last one occurred in January 2021, marking the bottom at around 90. The dollar caught the bid in the subsequent months, with the index eventually hitting a high of over 114.00 in September 2022. Note that price patterns do not always unfold as expected, meaning the impending death cross may not necessarily trap bears; however, being aware of its past tendency can help traders manage their positions more effectively. The dollar index, which tracks the greenback's value against major fiat currencies, tanked by 10.78% in the first half of the year, its worst performance since 1991.
