AI is killing the web. Can anything save it?

Mint

2 days ago

Around the beginning of last year, Matthew Prince started receiving worried calls from the chief executives of large media companies. They told Mr Prince, whose firm, Cloudflare, provides security infrastructure to about a fifth of the web, that their businesses faced a grave new online threat. 'I said, 'What, is it the North Koreans?'," he recalls. 'And they said, 'No. It's AI'."
Those executives had spotted the early signs of a trend that has since become clear: artificial intelligence is transforming the way that people navigate the web. As users pose their queries to chatbots rather than conventional search engines, they are given answers, rather than links to follow. The result is that 'content" publishers, from news providers and online forums to reference sites such as Wikipedia, are seeing alarming drops in their traffic.
As AI changes how people browse, it is altering the economic bargain at the heart of the internet. Human traffic has long been monetised using online advertising; now that traffic is drying up. Content producers are urgently trying to find new ways to make AI companies pay them for information. If they cannot, the open web may evolve into something very different.
Since the launch of ChatGPT in late 2022, people have embraced a new way to seek information online. OpenAI, maker of ChatGPT, says that around 800m people use the chatbot. It is the most popular download on the iPhone app store. Apple said that conventional searches in its Safari web browser had fallen for the first time in April, as people posed their questions to AI instead. OpenAI is soon expected to launch a browser of its own. Its rise is so dramatic that a Hollywood adaptation is in the works.
As OpenAI and other upstarts have soared, Google, which has about 90% of the conventional search market in America, has added AI features to its own search engine in a bid to keep up. Last year it began preceding some search results with AI-generated 'overviews", which have since become ubiquitous. In May it launched 'AI mode", a chatbot-like version of its search engine. The company promises that, with AI, users can 'let Google do the Googling for you".
Yet as Google does the Googling, humans no longer visit the websites from which the information is gleaned. Similarweb, which measures traffic to more than 100m web domains, estimates that worldwide search traffic (by humans) fell by about 15% in the year to June. Although some categories, such as hobbyists' sites, are doing fine, others have been hit hard (see chart). Many of the most affected are just the kind that might have commonly answered search queries. Science and education sites have lost 10% of their visitors. Reference sites have lost 15%. Health sites have lost 31%.
For companies that sell advertising or subscriptions, lost visitors means lost revenue. 'We had a very positive relationship with Google for a long time…They broke the deal," says Neil Vogel, head of Dotdash Meredith, which owns titles such as People and Food & Wine. Three years ago its sites got more than 60% of their traffic from Google. Now the figure is in the mid-30s. 'They are stealing our content to compete with us," says Mr Vogel. Google has insisted that its use of others' content is fair. But since it launched its AI overviews, the share of news-related searches resulting in no onward clicks has risen from 56% to 69%, estimates Similarweb. In other words, seven in ten people get their answer without visiting the page that supplied it.
'The nature of the internet has completely changed," says Prashanth Chandrasekar, chief executive of Stack Overflow, best known as an online forum for coders. 'AI is basically choking off traffic to most content sites," he says. With fewer visitors, Stack Overflow is seeing fewer questions posted on its message boards. Wikipedia, also powered by enthusiasts, warns that AI-generated summaries without attribution 'block pathways for people to access…and contribute to" the site.
To keep the traffic and the money coming, many big content producers have negotiated licensing deals with AI companies, backed up by legal threats: what Robert Thomson, chief executive of News Corp, has dubbed 'wooing and suing".His company, which owns theWall Street Journalandthe New York Post, among other titles, has struck a deal with OpenAI. Two of its subsidiaries are suing Perplexity, another AI answer engine. TheNew York Timeshas done a deal with Amazon while suing OpenAI. Plenty of other transactions and lawsuits are going on. (The Economist's parent company has not taken a public position on whether it will license our work.)
Yet this approach has limits. For one thing, judges so far seem minded to side with AI companies: last month two separate copyright cases in California went in favour of their defendants, Meta and Anthropic, both of which argued that training their models on others' content amounted to fair use. President Donald Trump seems to acceptSilicon Valley's argument that it must be allowed to get on with developing the technology of the future before China can. He has appointed tech boosters as advisers on AI, and sacked the head of the US Copyright Office soon after she argued that training AI on copyrighted material was not always legal.
AI companies are more willing to pay for continuing access to information than training data. But the deals done so far are hardly stellar. Reddit, an online forum, has licensed its user-generated content to Google for a reported $60m a year. Yet its market value fell by more than half—over $20bn—after it reported slower user-growth than expected in February, owing to wobbles in search traffic. (Growth has since picked up and Reddit's share price has recovered some lost ground.)
The bigger problem, however, is that most of the internet's hundreds of millions of domains are too small to either woo or sue the tech giants. Their content may be collectively essential to AI firms, but each site is individually dispensable. Even if they could join forces to bargain collectively, antitrust law would forbid it. They could block AI crawlers, and some do. But that means no search visibility at all.
Software providers may be able to help. All of Cloudflare's new customers will now be asked if they want to allow AI companies' bots to scrape their site, and for what purpose. Cloudflare's scale gives it a better chance than most of enabling something like a collective response by content sites that want to force AI firms to cough up. It is testing a pay-as-you-crawl system that would let sites charge bots an entry fee. 'We have to set the rules of the road," says Mr Prince, who says his preferred outcome is 'a world where humans get content for free, and bots pay a tonne for it".
An alternative is offered by Tollbit, which bills itself as a paywall for bots. It allows content sites to charge AI crawlers varying rates: for instance, a magazine could charge more for new stories than old ones. In the first quarter of this year Tollbit processed 15m micro-transactions of this sort, for 2,000 content producers including the Associated Pressand Newsweek. Toshit Panigrahi, its chief executive, points out that whereas traditional search engines incentivise samey content—'What time does the Super Bowl start?", for example—charging for access incentivises uniqueness. One of Tollbit's highest per-crawl rates is charged by a local newspaper.
Another model is being put forward by ProRata, a startup led by Bill Gross, a pioneer in the 1990s of the pay-as-you-click online ads that have powered much of the web ever since. He proposes that money from ads placed alongside AI-generated answers should be redistributed to sites in proportion to how much their content contributed to the answer. ProRata has its own answer engine,Gist.ai, which shares ad revenue with its 500-plus partners, which include theFinancial Timesand theAtlantic. It is currently more of an exemplar than a serious threat to Google: Mr Gross says his main aim is to 'show a fair business model that other people eventually copy".
Meanwhile, content producers are rethinking their business models. 'The future of the internet is not all about traffic," says Mr Chandrasekar, who has built up Stack Overflow's private, enterprise-oriented subscription product, Stack Internal. News publishers are planning for 'Google zero", deploying newsletters and apps to reach customers who no longer come to them via search, and moving their content behind paywalls or to live events. Audio and video are proving legally and technically harder for AI engines to summarise than text. The site to which answer engines refer search traffic most often, by far, is YouTube, according to Similarweb.
Not everyone thinks the web is in decline—on the contrary, it is in 'an incredibly expansionary moment", argues Robby Stein of Google. As AI makes it easier to create content, the number of sites is growing: Google's bots report that the web has expanded by 45% in the past two years. AI search lets people ask questions in new ways—for instance, taking a photo of their bookshelf and asking for recommendations on what to read next—which could increase traffic. With AI queries, more sites than ever are being 'read", even if not with human eyes. An answer engine may scan hundreds of pages to deliver an answer, drawing on a more diverse range of sources than human readers would.
As for the idea that Google is disseminating less human traffic than before, Mr Stein says the company has not noticed a dramatic decline in the number of outbound clicks, though it declines to make the number public. There are other reasons besides AI why people may be visiting sites less. Maybe they are scrolling social media. Maybe they are listening to podcasts.
The death of the web has been predicted before—at the hands of social networks, then smartphone apps—and not come to pass. But AI may pose the biggest threat to it yet. If the web is to continue in something close to its current form, sites will have to find new ways to get paid for content. 'There's no question that people prefer AI search," says Mr Gross. 'And to make the internet survive, to make democracy survive, to make content creators survive, AI search has to share revenue with creators."