Cloudflare launches tool to help website owners monetise AI bot crawler access
has launched a tool that blocks bot crawlers from accessing content without permission or compensation to help websites make money from AI firms trying to access and train on their content, the software company said on Tuesday.
The tool allows website owners to choose whether
artificial intelligence
crawlers can access their material and set a price for access through a "pay per crawl" model, which will help them control how their work is used and compensated, Cloudflare said.
With AI crawlers increasingly collecting content without sending visitors to the original source, website owners are looking to develop additional revenue sources as search traffic referrals that once generated advertising revenue decline.
The initiative is supported by major publishers including Conde Nast and Associated Press, as well as social media companies such as Reddit and Pinterest.
Cloudflare's Chief Strategy Officer Stephanie Cohen said the goal of such tools was to give publishers control over their content, and ensure a sustainable ecosystem for online content creators and
AI companies
.
"The change in traffic patterns has been rapid, and something needed to change," Cohen said in an interview. "This is just the beginning of a new model for the internet."
Google, for example, has seen its ratio of crawls to visitors referred back to sites drop to 18:1 from 6:1 just six months ago, according to Cloudflare data, suggesting the search giant is maintaining its crawling but decreasing referrals.
The decline could be a result of users finding answers directly within Google's search results, such as AI Overviews. Still, Google's ratio is much higher than other AI companies, such as OpenAI's 1,500:1.
For decades, search engines have indexed content on the internet directing users back to websites, an approach that rewards creators for producing quality content. However, AI companies' crawlers have disrupted this model because they harvest material without sending visitors to the original source and aggregate information through chatbots such as ChatGPT, depriving creators of revenue and recognition.
Many AI companies are circumventing a common web standard used by publishers to block the scraping of their content for use in AI systems, and argue they have broken no laws in accessing content for free.
In response, some publishers, including the New York Times, have sued AI companies for
copyright infringement
, while others have struck deals to license their content.
Reddit, for example, has sued AI startup Anthropic for allegedly scraping Reddit user comments to train its AI chatbot, while inking a
content licensing
deal with Google.
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Mint
30 minutes ago
- Mint
Soham Parekh, accused of ‘scamming' multiple companies, gets new job offer; recruiter say ‘he deserves a second chance'
Soham Parekh, an Indian techie who is at the centre of controversy on social media after a US-based startup founder accused him of working at different firms simultaneously - has received a job offer from an AI startup founder. Conor Brennan-Burke, the founder of the AI company HyperSpell, said that he believes in second chances and wrote: "He's definitely learned his lesson now and is going to work insanely hard to prove everyone wrong. Massive opportunity to bring on top talent with a chip on their shoulder." Burke also revealed the email he sent to Parekh, offering him an engineering position at his company. "Hi Soham, Saw that you've worked with quite a few successful startups and heard you might be on the market. We're hiring engineers right now. Let me know if you're open to a casual chat next week," the email read. Sharing the email, Burke remarked: "Be greedy when others are fearful." However, the AI founder faced warnings against hiring the Indian techie, with one user cautioning him about 'low-integrity people.' In response, he said, 'Everybody deserves a second chance. Let's be part of his redemption arc.' Viral post. Parekh has been accused by a US-based AI founder of holding jobs at several startups at the same time, allegedly under false claims. In a series of posts on X, Suhail Doshi, co-founder and former CEO of Mixpanel, labelled Soham Parekh a "scammer," claiming he misled multiple companies. Doshi said Parekh was briefly employed at one of his firms last year but was dismissed within the first week after his "scam" was discovered. He also mentioned that he had tried to warn Parekh about moonlighting, but the warning "clearly didn't work," as Parekh allegedly continued to work for several startups simultaneously. 'PSA: there's a guy named Soham Parekh (in India) who works at 3-4 startups at the same time. He's been preying on YC companies and more. Beware. I fired this guy in his first week and told him to stop lying/scamming people. He hasn't stopped a year later. No more excuses, 'Doshi, who also founded Playground AI, wrote. Several tech CEOs have alleged that Soham Parekh misled multiple US startups by accepting roles while continuing to work remotely from India. So far, over six company heads have shared their experiences, stating they had to terminate Parekh's employment after realising he was 'moonlighting.' According to the CV shared by Doshi, Soham Parekh holds a bachelor's degree from the University of Mumbai and a master's from the Georgia Institute of Technology. His résumé lists stints at several tech firms, including Dynamo AI, Union AI, Synthesia, and Alan AI — mostly in engineering and technical roles.
Time of India
41 minutes ago
- Time of India
Chinese AI startup that 'shocked' biggest American tech companies is hiring big time
DeepSeek , the Chinese artificial intelligence (AI) startup based in Hangzhou, appears to be expanding its talent search beyond its borders, significantly increasing its recruitment efforts on LinkedIn. A report says that over the past week, the company posted 10 new positions on the Microsoft-owned professional networking platform, marking its first listings there in several months. The job postings, written in Mandarin with detailed descriptions, include three roles specifically focused on Artificial General Intelligence (AGI). These positions are based in DeepSeek's Beijing and Hangzhou offices. According to a report by Bloomberg, similar roles were posted on popular Chinese recruitment sites earlier in 2025, the increased activity on LinkedIn is notable. LinkedIn ceased operating a localised version of its platform in China in 2021, meaning many candidates viewing these listings would likely be based outside of the country. Tech companies like Google, OpenAI and Meta increase 'AI hiring' DeepSeek's move to attract global talent mirrors the aggressive recruitment strategies of its US rivals, such as Google , OpenAI (developer of ChatGPT) and Meta Platforms. These companies are actively competing to secure top AI experts in the race to dominate the potentially world-changing field of artificial intelligence. Recently, Meta CEO Mark Zuckerberg introduced the team that will lead the company's 'Superintelligence' lab. The team consists of former employees of companies like OpenAI, Google and Anthropic, and will be lead by ScaleAI founder Alexandr Wang and former GitHub CEO Nat Friedman. Last year, Microsoft hired Mustafa Suleyman, co-founder of DeepMind – now part of Google – and Inflection, as CEO of Microsoft AI team. Elon Musk has, on several occasions, posted job opportunities for AI engineers to work in his AI comapby, xAI. OnePlus Nord 5 and OnePlus Nord CE 5: Unboxing and first look AI Masterclass for Students. Upskill Young Ones Today!– Join Now
Time of India
42 minutes ago
- Time of India
‘Running a classroom, not a company': Startup founder reveals the harsh realities of building a business in India
A pointed LinkedIn post by Sanket S, founder of Scandolous Foods, has sent ripples through India's startup ecosystem, sparking a flurry of reactions from entrepreneurs and professionals who say it highlights deep flaws in the country's talent pipeline. Check full text of the post here I hired 3 people from one of India's best tech colleges and honestly…it scared me. I'm not talking about some no-name college. These kids paid ₹40–50L for degrees from India's top private MBA, food, and hospitality colleges. But they walked out knowing… nothing that actually matters. No idea what precision fermentation is. MBA grad didn't understand P&L or cash flow. Hotel management kid had never seen a food processing line. And all of them are brilliant at making PPTs. That too stuff Gemini or ChatGPT can do in seconds now. What are we training kids for?? Because it's clearly not for work. It's to memorise outdated textbooks and polish case studies from 2012. And I'm sitting here, trying to build a 'globally competitive company' and this is the talent I get access to? What am I supposed to do? Train them from scratch, then I'm not running a company, I'm running a classroom. Or hire from abroad and feel like a traitor to my own 'Make in India' dreams? Honestly, I've spoken to so many founders, and this is not just MY problem. This is India's problem. And it pisses me off because founders can hustle, investors can bet big but if the talent pipeline is broken, the whole system crumbles. You want India to lead in food tech, biotech, climate tech, medtech? Then stop producing talent that's outdated before it even hits the job market. Because at this rate, we're not just 10 years behind, we're raising a generation that doesn't even know what the world looks like today. What did people say? "Yes it's every founder's problem. Need to train them from scratch and no guarantee that they stay with the company and can quit at any time. There's a huge gap in the education system which needs to be addressed immediately," said one user. "The situation is alarming. Sanket S you have raised a very valid point. In fact, many of us as an academician also find it in the same way. Institutes compete against peers in showcasing their highest placement packages; Similarly, students also want to be assured of getting a good placement and hence evaluates the institute on this one and only criterion. In this kind of transactional approach, intellectual resource, skill building, and many more things of utmost importance takes the back and many a times it results into something that disappoints practitioners like you at the end," said another user.
