Latest news with #RogerLynch
Indian Express
02-07-2025
- Business
- Indian Express
India must act: Publishers sound alarm on AI's ‘theft' of news content
As leading publishers in the US and UK clamp down on artificial intelligence (AI) companies scraping news content without consent, Indian digital media houses are ramping up pressure on the government to step in and protect journalistic work from being 'exploited' by commercial AI models. The development follows a sweeping move by major global players, including the Associated Press, The Atlantic, Sky News, Time, Buzzfeed, Conde Nast, and DMGT, to block AI bots from crawling their websites by default. The effort is being supported by Cloudflare, one of the world's largest internet infrastructure firms, which has announced a new system that gives publishers granular control over AI access to their sites. Indian publishers say the problem of unauthorised AI scraping has reached alarming proportions with no legal safeguards, licensing systems, or enforcement mechanisms in place. 'The situation in India is becoming increasingly untenable,' said a spokesperson for the Digital News Publishers Association (DNPA), a key industry body. 'While global players are waking up to the importance of permission and fair compensation, Indian news content continues to be freely exploited without dialogue or safeguards. We urge the Government of India to take immediate steps to take necessary measures against such unauthorised and rampant data scrapping,' the spokersperson said in a press statement. According to Cloudflare, OpenAI's GPTBot alone accounted for nearly 30 per cent of all AI-related web scraping in May 2025, a sixfold increase from just a year earlier. Other major scrapers include Meta's External Agent and Anthropic's ClaudeBot. Cloudflare's new approach gives publishers the option to allow or block AI crawlers and even tag them based on intent, whether the bot is scraping for model training, search indexing, or inference purposes. Crucially, the company is also testing a 'pay-per-crawl' model that could allow news outlets to charge AI firms directly for accessing their content. 'This is a game-changer for publishers,' said Roger Lynch, CEO of Conde Nast. 'When AI companies can no longer take anything they want for free, it opens the door to sustainable innovation built on permission and partnership.' But in India, publishers have few tools at their disposal. Unlike in the US and UK, where AI companies now face increasing legal scrutiny and regulatory pushback, Indian media houses are still operating in a grey zone. The DNPA and other digital publishers have now jointly called on the Ministry of Electronics and IT (MeitY) and the Ministry of Information and Broadcasting (MIB) to urgently act on the following demands: 'India has the potential to be a global AI leader – but not by trampling on the rights of its own creators,' said a senior editor from a leading digital news platform. 'We must innovate responsibly, with laws that value original content and protect public trust.'
Tahawul Tech
02-07-2025
- Business
- Tahawul Tech
Cloudflare introduces AI bot blocker
Millions of websites will now be able to block AI bots from accessing their content without permission thanks to a new system being rolled out by internet infrastructure firm, Cloudflare. Eventually, sites will be able to ask for payment from AI firms in return for having their content scraped. Cloudflare's tech targets AI firm bots – also known as crawlers – which are programs that explore the web, indexing and collecting data as they go. They are important to the way AI firms build, train and operate their systems. So far, Cloudflare says its tech is active on a million websites. Roger Lynch, chief executive of Condé Nast, whose print titles include GQ, Vogue, and The New Yorker, said the move was 'a game-changer' for publishers. 'This is a critical step toward creating a fair value exchange on the Internet that protects creators, supports quality journalism and holds AI companies accountable', he wrote in a statement. However, other experts say stronger legal protections will still be needed. Initially the system will apply by default to new users of Cloudflare services, plus sites that participated in an earlier effort to block crawlers. Many publishers accuse AI firms of using their content without permission. Cloudflare argues AI breaks the unwritten agreement between publishers and crawlers. AI crawlers, it argues, collect content like text, articles, and images to generate answers, without sending visitors to the original source—depriving content creators of revenue. 'If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone,' wrote the firm's chief executive Matthew Prince. To that end the company is developing a 'Pay Per Crawl' system, which would give content creators the option to request payment from AI companies for utilising their original content. According to Cloudflare there has been an explosion of AI bot activity. 'AI Crawlers generate more than 50 billion requests to the Cloudflare network every day', the company wrote in March. And there is growing concern that some AI crawlers are disregarding existing protocols for excluding bots. In an effort to counter the worst offenders Cloudflare previously developed a system where the worst miscreants would be sent to a 'Labyrinth' of web pages filled with AI generated junk. The new system attempts to use technology to protect the content of websites and to give sites the option to charge AI firms a fee to access it. Source: BBC News Image Credit: Cloudflare
Techday NZ
02-07-2025
- Business
- Techday NZ
Cloudflare makes AI crawlers opt-in, giving power to creators
Cloudflare has introduced a default setting to block AI crawlers from accessing web content without explicit permission, making it the first internet infrastructure provider to take this step. With this new measure, website owners using Cloudflare's services will have the choice to allow or block AI crawlers, moving from a previous opt-out system to an opt-in approach. This change is designed to address issues concerning the unauthorised scraping and usage of web content by AI companies for purposes such as training and inference, often without the knowledge or compensation of the content creators. Permission-based controls Under the new system, AI companies are now required to disclose the purpose of their crawlers, specifying whether they are used for training, inference, or search. This allows website owners to make more informed decisions about which bots may access their data. Cloudflare is also developing a "Pay Per Crawl" feature that will give content creators the ability to request payment from AI companies for access to their content, which could generate new revenue streams for publishers. Cloudflare's Chief Executive Officer and Co-founder, Matthew Prince, stated: "If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone – creators, consumers, tomorrow's AI founders, and the future of the web itself. Original content is what makes the Internet one of the greatest inventions in the last century, and it's essential that creators continue making it. AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone." This revised approach follows previous Cloudflare initiatives to block AI crawlers, which began with a one-click option introduced in September 2024. Since then, more than one million customers have chosen to restrict AI crawlers from their websites. Now, blocking occurs by default for all new customers, eliminating the need for domain owners to adjust settings to prevent unauthorised crawling. Support from publishers Prominent media organisations and publishers have expressed support for Cloudflare's move, including ADWEEK, SkyNews, Fortune, The Associated Press, BuzzFeed, The Atlantic, TIME, Reddit, and Pinterest. These companies have advocated for fair compensation frameworks and greater transparency around how content is accessed and used by AI platforms. Roger Lynch, Chief Executive Officer of Condé Nast, commented: "Cloudflare's innovative approach to block AI crawlers is a game-changer for publishers and sets a new standard for how content is respected online. When AI companies can no longer take anything they want for free, it opens the door to sustainable innovation built on permission and partnership. This is a critical step toward creating a fair value exchange on the Internet that protects creators, supports quality journalism and holds AI companies accountable." Neil Vogel, Chief Executive Officer of Dotdash Meredith, added: "We have long said that AI platforms must fairly compensate publishers and creators to use our content. We can now limit access to our content to those AI partners willing to engage in fair arrangements. We're proud to support Cloudflare and look forward to using their tools to protect our content and the open web." Renn Turiano, Chief Consumer and Product Officer at Gannett Media, also noted: "As the largest publisher in the country, comprised of USA TODAY and over 200 local publications throughout the USA TODAY Network, blocking unauthorised scraping and the use of our original content without fair compensation is critically important. As our industry faces these challenges, we are optimistic the Cloudflare technology will help combat the theft of valuable IP." Bill Ready, Chief Executive Officer of Pinterest, said: "Creators and publishers around the world leverage Pinterest to expand their businesses, reach new audiences and directly measure their success. As AI continues to reshape the digital landscape, we are committed to building a healthy Internet infrastructure where content is used for its intended purpose, so creators and publishers can thrive." Steve Huffman, Co-founder and Chief Executive Officer of Reddit, stated: "AI companies, search engines, researchers, and anyone else crawling sites have to be who they say they are. And any platform on the web should have a say in who is taking their content for what. The whole ecosystem of creators, platforms, web users and crawlers will be better when crawling is more transparent and controlled, and Cloudflare's efforts are a step in the right direction for everyone." Vivek Shah, Chief Executive Officer of Ziff Davis, commented: "We applaud Cloudflare for advocating for a sustainable digital ecosystem that benefits all stakeholders — the consumers who rely on credible information, the publishers who invest in its creation, and the advertisers who support its dissemination." Industry consortia and authentication Cloudflare is also participating in the development of new technical protocols to allow AI bots to authenticate themselves and for website owners to reliably determine the identity and intent of incoming requests. This aims to improve overall transparency and control over the use of web content by automated agents. Additional media and technology companies have added their support, indicating a broad industry move towards permission-based AI access to digital content. The list includes companies such as The Arena Group, Atlas Obscura, Quora, Stack Overflow, Universal Music Group, O'Reilly Media, and others. This change comes as publishers report reduced website traffic and declining advertising revenues linked to AI platforms generating answers directly to user queries without referencing or referring traffic to the original sources. Cloudflare's new default blocking of AI crawlers aims to restore a value exchange between content creators, consumers, and technology companies as artificial intelligence continues to shape the internet landscape.
BBC News
01-07-2025
- Business
- BBC News
Millions of websites to get 'game-changing' AI bot blocker
Millions of websites - including Sky News, The Associated Press and Buzzfeed - will now be able to block artificial intelligence (AI) bots from accessing their content without new system is being rolled out by internet infrastructure firm, Cloudflare, which hosts around a fifth of the internet. Eventually, sites will be able to demand payment from AI firms in return for having their content prominent writers, artists, musicians and actors have accused AI firms of training systems on their work without permission or the UK, it led to a furious row between the government and artists including Sir Elton John over how to protect copyright. Cloudflare's tech targets AI firm bots - also known as crawlers - programmes that explore the web, indexing and collecting data as they go. They are important to the way AI firms build, train and operate their far, Cloudflare says its tech is active on a million Lynch, chief executive of Condé Nast, whose print titles include GQ, Vogue, and The New Yorker, said the move was "a game-changer" for publishers."This is a critical step toward creating a fair value exchange on the Internet that protects creators, supports quality journalism and holds AI companies accountable", he wrote in a other experts say stronger legal protections will still be needed. 'Surviving the age of AI' Initially the system will apply by default to new users of Cloudflare services, plus sites that participated in an earlier effort to block publishers accuse AI firms of using their content without the BBC threatened to take legal action against US based AI firm Perplexity, demanding it immediately stopped using BBC content, and paid compensation for material already publishers are generally happy to allow crawlers from search engines, like Google, to access their sites, so that the search companies can in return can direct people to their content. Perplexity accused the BBC of seeking to preserve "Google's monopoly". But Cloudflare argues AI breaks the unwritten agreement between publishers and crawlers. AI crawlers, it argues, collect content like text, articles, and images to generate answers, without sending visitors to the original source—depriving content creators of revenue. "If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone," wrote the firm's chief executive Matthew Prince. To that end the company is developing a "Pay Per Crawl" system, which would give content creators the option to request payment from AI companies for utilising their original content. Battle the bots According to Cloudflare there has been an explosion of AI bot activity. "AI Crawlers generate more than 50 billion requests to the Cloudflare network every day", the company wrote in there is growing concern that some AI crawlers are disregarding existing protocols for excluding an effort to counter the worst offenders Cloudflare previously developed a system where the worst miscreants would be sent to a "Labyrinth" of web pages filled with AI generated new system attempts to use technology to protect the content of websites and to charge AI firms to access the UK there is an intense legislative battle between government, creators and the AI firms over the extent to which the creative industries should be protected from AI firms using their works to train systems without permission or on both sides of the Atlantic, content creators, licensors and owners have gone to court in an effort to prevent what they see as AI firms encroachment on creative Newton-Rex, the founder of Fairly Trained which certifies that AI companies have trained their systems on properly licensed data, said it was a welcome development - but there was "only so much" one company could do "This is really only a sticking plaster when what's required is major surgery," he told the BBC."It will only offer protection for people on websites they control - it's like having body armour that stops working when you leave your house," he added."The only real way to protect people's content from theft by AI companies is through the law."
Techday NZ
01-07-2025
- Business
- Techday NZ
Cloudflare blocks AI crawlers to support content creators
Cloudflare has implemented a new default setting to block AI crawlers from accessing website content without explicit permission or compensation, making it the first internet infrastructure provider to do so. With this change, every new customer and domain on Cloudflare's platform will start with a setting that blocks AI crawlers by default, shifting the responsibility to AI companies to request access and clarify the crawler's intended purpose, such as training, inference, or search. This new approach replaces the previous opt-out system with an opt-in model, giving more power to content creators and publishers over the use of their work. Cloudflare is also developing a feature called "Pay Per Crawl," which would allow content creators to request payment from AI companies seeking to use their content, thereby creating potential new revenue streams. This move addresses concerns about AI companies scraping web content without consent or compensation—a practice that many publishers and stakeholders argue threatens the future economic sustainability of the internet. Shifting value in online content The longstanding model of the internet has been based on a cycle in which search engines index web content, drive traffic to original websites, and provide revenue to creators through advertising. However, according to Cloudflare, the growing use of AI crawlers that extract information for large language models and other generative applications has disrupted this cycle by delivering answers without redirecting users to the original source. This change means creators may lose both the financial benefits and audience engagement historically generated by their work. Matthew Prince, Cloudflare's Co-founder and CEO, commented, "If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone – creators, consumers, tomorrow's AI founders, and the future of the web itself. Original content is what makes the Internet one of the greatest inventions in the last century, and it's essential that creators continue making it. AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone." This sentiment has been echoed by several publishers and content platforms. Roger Lynch, CEO of Condé Nast, stated, "Cloudflare's innovative approach to block AI crawlers is a game-changer for publishers and sets a new standard for how content is respected online. When AI companies can no longer take anything, they want for free, it opens the door to sustainable innovation built on permission and partnership. This is a critical step toward creating a fair value exchange on the Internet that protects creators, supports quality journalism and holds AI companies accountable." Neil Vogel, CEO of Dotdash Meredith, remarked, "We have long said that AI platforms must fairly compensate publishers and creators to use our content. We can now limit access to our content to those AI partners willing to engage in fair arrangements. We're proud to support Cloudflare and look forward to using their tools to protect our content and the open web." Renn Turiano, Chief Consumer and Product Officer at Gannett Media, noted, "As the largest publisher in the country, comprised of USA TODAY and over 200 local publications throughout the USA TODAY Network, blocking unauthorised scraping and the use of our original content without fair compensation is critically important. As our industry faces these challenges, we are optimistic the Cloudflare technology will help combat the theft of valuable IP." Other technology companies have also spoken in support of the new permission-based system. Bill Ready, CEO of Pinterest, stated, "Creators and publishers around the world leverage Pinterest to expand their businesses, reach new audiences and directly measure their success. As AI continues to reshape the digital landscape, we are committed to building a healthy Internet infrastructure where content is used for its intended purpose, so creators and publishers can thrive." Steve Huffman, Reddit's Co-founder and CEO, pointed out, "AI companies, search engines, researchers, and anyone else crawling sites have to be who they say they are. And any platform on the web should have a say in who is taking their content for what. The whole ecosystem of creators, platforms, web users and crawlers will be better when crawling is more transparent and controlled, and Cloudflare's efforts are a step in the right direction for everyone." Vivek Shah, CEO of Ziff Davis, added, "We applaud Cloudflare for advocating for a sustainable digital ecosystem that benefits all stakeholders — the consumers who rely on credible information, the publishers who invest in its creation, and the advertisers who support its dissemination." Default enforcement Prior to this change, Cloudflare had already offered a one-click option to block AI crawlers since mid-2024. Over a million customers have enabled this option. With the latest move, every new website signing up to Cloudflare will be prompted to decide whether to allow or deny AI crawler access, streamlining the decision process and ensuring the default favours content owner control. Industry support More than 30 publishers, media, and technology companies have voiced their support for the new permission-based crawling model, including ADWEEK, The Associated Press, TIME, The Atlantic, Reddit, Pinterest, Quora, Sky News Group, and Universal Music Group, among others. Will Lee, CEO of ADWEEK, stated, "As the front page and homepage for marketing, advertising and media industry leaders, ADWEEK's position has been clear that we must be compensated for our investment grade journalism and information. I am thrilled Cloudflare has created a marketplace and mechanism that will enable us to properly participate in the promise LLMs have for our industry." Paul Edmondson, CEO of The Arena Group, said, "We think of our writers and content creators as entrepreneurs. Their work deserves protection. By blocking unauthorized AI crawlers, Cloudflare is not just defending content – it's defending the future of creators and storytellers. This is a vital move toward a digital economy built on trust, permission and fair value." Additional key supporters include BuzzFeed, PMC, Quora, Stack Overflow, News/Media Alliance, and Webflow, all of whom have commented on the significance of the move for the digital economy and the rights of creators. Technical implementation Cloudflare has also put forward new ways for AI bots to authenticate themselves and for webmasters to identify them, including participating in the development of industry protocols for bot identification and authentication. These mechanisms aim to increase transparency in the operation of AI crawlers, allowing website owners to make more informed choices about access to their content. AI companies are now required to obtain clear, explicit permission from websites prior to scraping content for AI training or generation purposes. Existing and future customers of Cloudflare can review and modify their crawler settings as required.
