Latest news with #webcrawlers
Tahawul Tech
02-07-2025
- Business
- Tahawul Tech
Cloudflare introduces AI bot blocker
Millions of websites will now be able to block AI bots from accessing their content without permission thanks to a new system being rolled out by internet infrastructure firm, Cloudflare. Eventually, sites will be able to ask for payment from AI firms in return for having their content scraped. Cloudflare's tech targets AI firm bots – also known as crawlers – which are programs that explore the web, indexing and collecting data as they go. They are important to the way AI firms build, train and operate their systems. So far, Cloudflare says its tech is active on a million websites. Roger Lynch, chief executive of Condé Nast, whose print titles include GQ, Vogue, and The New Yorker, said the move was 'a game-changer' for publishers. 'This is a critical step toward creating a fair value exchange on the Internet that protects creators, supports quality journalism and holds AI companies accountable', he wrote in a statement. However, other experts say stronger legal protections will still be needed. Initially the system will apply by default to new users of Cloudflare services, plus sites that participated in an earlier effort to block crawlers. Many publishers accuse AI firms of using their content without permission. Cloudflare argues AI breaks the unwritten agreement between publishers and crawlers. AI crawlers, it argues, collect content like text, articles, and images to generate answers, without sending visitors to the original source—depriving content creators of revenue. 'If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone,' wrote the firm's chief executive Matthew Prince. To that end the company is developing a 'Pay Per Crawl' system, which would give content creators the option to request payment from AI companies for utilising their original content. According to Cloudflare there has been an explosion of AI bot activity. 'AI Crawlers generate more than 50 billion requests to the Cloudflare network every day', the company wrote in March. And there is growing concern that some AI crawlers are disregarding existing protocols for excluding bots. In an effort to counter the worst offenders Cloudflare previously developed a system where the worst miscreants would be sent to a 'Labyrinth' of web pages filled with AI generated junk. The new system attempts to use technology to protect the content of websites and to give sites the option to charge AI firms a fee to access it. Source: BBC News Image Credit: Cloudflare
Yahoo
02-07-2025
- Business
- Yahoo
Internet infrastructure company Cloudflare ‘changes the rules of the internet' by blocking AI crawlers
Cloudflare, the internet infrastructure company that powers much of the web, says that it is 'changing the rules of the internet'. The company will block AI crawlers by default in an attempt to stop artificial intelligence companies from gathering data from across the web. Large language models, such as ChatGPT, require vast amounts of training data in order to improve their performance. Many of those companies have responded by using crawlers to access different web pages and store their contents so that it can be used to train those systems. Now, Cloudflare says that those AI crawlers will be blocked by default. That means those automated systems should not be able to visit pages without 'permission or compensation', it said. The publishers of those websites will be able to give AI crawlers permission to access their sites. And Cloudflare says that it will add the option for a 'pay by crawl' fee, which AI companies will be able to choose whether to pay. The use of online writing to train AI systems has become one of the most divisive issues of the ongoing era of artificial intelligence. A number of publishers have become locked in legal action against AI companies that they argue have wrongly taken their writing to train their systems. As that conflict increased, in 2023, Cloudflare initially said that it would allow websites to add a special tag to their sites to ask AI websites not to access them, but it was unenforceable. Last year, it started to allow websites to automatically block them, using technology that spotted such AI bots. Now, that latter tool will be on by default. Matthew Prince, Cloudflare's chief executive, told the New York Times that it was 'changing the rules of the internet across all of Cloudflare'. The move was made from a concern that AI companies scraping the web and accessing content freely would stop people being incentivised from publishing new writing or other forms of content, he said.
The Verge
01-07-2025
- Business
- The Verge
Cloudflare will now block AI crawlers by default
The major internet architecture provider Cloudflare will now block known AI web crawlers by default to prevent them from 'accessing content without permission or compensation,' according to an announcement on Tuesday. With the change, Cloudflare will start asking new domain owners whether they want to allow AI scrapers, and will even let some publishers implement a 'Pay Per Crawl' fee. The Pay Per Crawl program will let publishers set a price for AI scrapers to access their content. AI companies can then view pricing and choose whether to register for the 'Pay Per Crawl' fee or turn away. This is only available for 'a group of some of the leading publishers and content creators' for now, but Cloudflare says it will ensure 'AI companies can use quality content the right way — with permission and compensation.' Cloudflare has been helping domain owners fight AI crawlers for a while now. The company started letting websites block AI crawlers in 2023, but it only applied to ones that abide by a site's file, the unenforceable agreement that signals whether bots can scrape its content. Cloudflare began allowing websites to block 'all' AI bots last year — whether they respect a site's file or not — and now this setting is enabled by default for new Cloudflare customers. (The company identifies scrapers to block by comparing them to its list of known AI bots.) Cloudflare also rolled out a feature in March that sends web-crawling bots into an 'AI Labyrinth' to deter them from scraping sites without permission. Several major publishers and online platforms, including The Associated Press, The Atlantic, Fortune, Stack Overflow, and Quora, are on board with Cloudflare's new AI crawler restrictions, as websites contend with a future where more people are finding information through AI chatbots, rather than search engines. 'People trust the AI more over the last six months, which means they're not reading original content,' Cloudflare CEO Matthew Prince said during the Axios Live event last week. Additionally, Cloudflare says it's working with AI companies to help verify their crawlers and allow them to 'clearly state their purpose,' such as whether they're using the content for training, inference, or search. Website owners can then review this information and determine which crawlers to let in. 'Original content is what makes the Internet one of the greatest inventions in the last century, and we have to come together to protect it,' Prince said in the press release. 'AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate.'
