‘This is coming for everyone': A new kind of AI bot takes over the web
To offer users a tidy AI summary instead of Google's '10 blue links,' companies such as OpenAI and Anthropic have started sending out bots to retrieve and recap content in real time. They are scraping webpages and loading relevant content into the AI's memory and 'reading' far more content than a human ever would.
Subscribe to The Post Most newsletter for the most important and interesting stories from The Washington Post.
According to data shared exclusively with The Washington Post, traffic from retrieval bots grew 49 percent in the first quarter of 2025 from the fourth quarter of 2024. The data is from TollBit, a New York-based start-up that helps news publishers monitor and make money when AI companies use their content.
TollBit's report, based on data from 266 websites - half of which are run by national and local news organizations - suggests that the growth of bots that retrieve information when a user prompts an AI model is on an exponential curve.
'It starts with publishers, but this is coming for everyone,' Toshit Panigrahi, CEO and co-founder of TollBit, said in an interview.
Panigrahi said that this kind of bot traffic, which can be hard for websites to detect, reflects growing demand for content, even as AI tools devastate traffic to news sites and other online platforms. 'Human eyeballs to your site decreased. But the net amount of content access, we believe, fundamentally is going to explode,' he said.
A spokesperson for OpenAI said that referral traffic to publishers from ChatGPT searches may be lower in quantity but that it reflects a stronger user intent compared with casual web browsing.
To capitalize on this shift, websites will need to reorient themselves to AI visitors rather than human ones, Panigrahi said. But he also acknowledged that squeezing payment for content when AI companies argue that scraping online data is fair use will be an uphill climb, especially as leading players make their newest AI visitors even harder to identify.
Debate around the AI industry's use of online content has centered on the gargantuan amounts of text needed to train the AI models that power tools like ChatGPT. To obtain that data, tech companies use bots that scrape the open web for free, which has led to a raft of lawsuits alleging copyright theft from book authors and media companies, including a New York Times lawsuit against OpenAI. Other news publishers have opted for licensing deals. (In April, The Washington Post inked a deal with OpenAI.)
In the past eight months, as chatbots have evolved to incorporate features like web search and 'reasoning' to answer more complex queries, traffic for retrieval bots has skyrocketed. It grew 2.5 times as fast as traffic for bots that scrape data for training between the fourth quarter of 2024 and the first quarter of 2025, according to TollBit's report.
Panigrahi said TollBit's data may underestimate the magnitude of this change because it doesn't reflect bots that AI companies send out on behalf of AI 'agents' that can complete tasks on a user's behalf, like ordering takeout from DoorDash.
The start-up's findings also add a new dimension to mounting evidence that the modern internet - optimized for Google search results and social media algorithms - will have to be restructured as the popularity of AI answers grows.
'To think of it as, 'Well, I'm optimizing my search for humans' is missing out on a big opportunity,' he said.
Installing TollBit's analytics platform is free for news publishers, and the company has more than 2,000 clients, many of which are struggling with these seismic changes, according to data in the report.
Although news publishers and other websites can implement blockers to prevent various AI bots from scraping their content, TollBit found that more than 26 million AI scrapes bypassed those blockers in March alone. Some AI companies claim bots for AI agents don't need to follow bot instructions because they are acting on behalf of a user.
Mark Howard, chief operating officer for the media company Time, a TollBit client, said the start-up's traffic data has helped Time negotiate content licensing deals with AI companies including OpenAI and the search engine Perplexity.
But the market to fairly compensate publishers is far from established, Howard said. 'The vast majority of the AI bots out there absolutely are not sourcing the content through any kind of paid mechanism. … There is a very, very long way to go.'
Related Content
Field notes from the end of life: My thoughts on living while dying
He's dying. She's pregnant. His one last wish is to fight his cancer long enough to see his baby.
The U.S. granted these journalists asylum. Then it fired them.
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Yahoo
7 minutes ago
- Yahoo
CEO Tim Cook says Apple ready to open its wallet to catch up in AI
By Stephen Nellis SAN FRANCISCO (Reuters) -Apple CEO Tim Cook signaled on Thursday the iPhone maker was ready to spend more to catch up to rivals in artificial intelligence by building more data centers or buying a larger player in the segment, a departure from a long practice of fiscal frugality. Apple has struggled to keep pace with rivals such as Microsoft and Alphabet's Google, both of which have attracted hundreds of millions of users to their AI-powered chatbots and assistants. That growth has come at a steep cost, however, with Google planning to spend $85 billion over the next year and Microsoft on track to spend more than $100 billion, mostly on data centers. Apple, in contrast, has leaned on outside data center providers to handle some of its cloud computing work, and despite a high-profile partnership with ChatGPT creator OpenAI for certain iPhone features, has tried to grow much of its AI technology in-house, including improvements to its Siri virtual assistant. The results have been rocky, with the company delaying its Siri improvements until next year. During a conference call after Apple's fiscal third-quarter results, analysts noted that Apple has historically not done large deals and asked whether it might take a different approach to pursue its AI ambitions. CEO Cook responded that the company had already acquired seven smaller companies this year and is open to buying larger ones. "We're very open to M&A that accelerates our roadmap. We are not stuck on a certain size company, although the ones that we have acquired thus far this year are small in nature," Cook said. "We basically ask ourselves whether a company can help us accelerate a roadmap, and if they do, then we're interested." Apple has tended to buy smaller firms with highly specialized technical teams to build out specific products. Its largest deal ever was its purchase of Beats Electronics for $3 billion in 2014, followed by a $1 billion deal to buy a modem chip business from Intel. But now Apple is at a unique crossroads for its business. The tens of billions of dollars per year it receives from Google as payment to be the default search engine on iPhones could be undone by U.S. courts in Google's antitrust trial, while startups like Perplexity are in discussions with handset makers to try to dislodge Google with an AI-powered browser that would handle many search functions. Apple executives have said in court they are considering reshaping the firm's Safari browser with AI-powered search functions, and Bloomberg News has reported that Apple executives have discussed buying Perplexity, which Reuters has not independently confirmed. Apple also said on Thursday it plans to spend more on data centers, an area where it typically spends only a few billion dollars per year. Apple is currently using its own chip designs to handle AI requests with privacy controls that are compatible with the privacy features on its devices. Kevan Parekh, Apple's chief financial officer, did not give specific spending targets but said outlays would rise. "It's not going to be exponential growth, but it is going to grow substantially," Parekh said during the conference call. "A lot of that's a function of the investments we're making in AI." Sign in to access your portfolio
Yahoo
7 minutes ago
- Yahoo
Hub Group is fully behind a potential UP-NS transcontinental railroad creation
Hub Group, a major intermodal transportation provider and a company that would be on the front lines of a merger between Union Pacific and Norfolk Southern, likes what it sees. In the prepared statement released in conjunction with the company's second quarter earnings, Hub Group (NASDAQ: HUBG), said it was 'supportive' of the two companies and their merger plans to create the nation's first transcontinental railroad. . 'The announced transaction would further accelerate our long-term growth opportunity,' the company said. 'Specifically, a transcontinental network removes friction in gateways, reduces transit times, provides access to new markets, and increases competition with truck volume through new single-line service.' Can we talk about something else? When President and CEO Phil Yeager first brought up the merger on the company's call with analysts Thursday after the numbers and statement were released, he echoed the positive sentiments in the earnings release. But he also asked the analysts on the call to not focus on the merger and the impact it might have on Hub Group. 'We would appreciate questions being focused on the company and our results,' Yeager said. And then when the phone lines were open to questions, Yeager's request was promptly ignored and the first question was about the merger. Despite his earlier admonition, Yeager took the question and continued to express support for the creation of the cross-country rail supergiant. Yeager, in his prepared remarks, said Hub Group were 'exclusive partners' with both Union Pacific and Norfolk Southern. With the two of them together, he said, 'there are several catalysts that should create significant intermodal conversion,' citing 'improved fluidity' in gateway cities, 'faster transit, better asset utilization, enhanced fuel efficiency and access to additional lanes and markets.' Yeager said about 30% of the Hub Group's current business is 'moving in a transcontinental fashion.' But since there is no single transcontinental railroad–establishing one being the point of the Union Pacific-Norfolk Southern (NYSE: UNP) (NYSE: NSC) tieup–Yeager expressed optimism about the efficiencies that could come from the existence of such a system instead of needing transfers between regional railroads.. Yeager said on the call that the transcontinental business is 'typically a positive mix' for both revenue and margins. Optimism on the Marten acquisition Hub Group's earnings release was the first since it announced its plan to acquire the intermodal operations of Marten Transport (NASDAQ: MRTN). Yeager said the acquisition 'allows us to enhance our scale and capacity in one of the highest growth segments of our intermodal network,' which is refrigerated. While the Marten intermodal operations had been consistently running with an operating ratio in excess of 100% for several quarters, Yeager said he believes the operations inside Hub Group will ''expand our customer base while generating strong returns, due to our ability to capture synergies within our platform.' As for more purchases, Yeager said Hub Group has a 'robust pipeline of additional acquisitions designed to continue deploying capital toward long term growth opportunities.' Kevin Beth, the company's CFO, after reviewing a decidedly mixed and not overly optimistic outlook on the state of the freight business for the rest of the year, did note one sign of strength for rail transportation. 'It's very positive that we're seeing peak season surcharges in July, and we hopefully will see that momentum carried forward in August and September and through the remainder of the year as well,' Beth said. Beth, in response to an analyst question, said the surcharges Hub Group has seen in the market this year are larger than last year, but they also were implemented by railroads later this year than in 2024. Yeager said the company anticipates an early West Coast peak season as part of an 'inventory pull forward' driven by importers trying to get ahead of tariffs. But specific to Hub Group, he said, the company has had an 'improved bid realization rate' and has added several new dedicated customers, 'which should lead to higher revenue from current levels.' The company's second earnings reported that several financial measures were weaker for Hub Group in the quarter. Operating income declined 13.1% from the second quarter of 2024 to $34.3 million. Net income dropped 13.7% to just over $25 million. The cost of purchased transportation at Hub Group, which is easily the largest expense at the company, declined 9.8%. It accounted for 72.4% of all operating expenses, down from 73.7% a year earlier. Specific units of the company both reported significant declines in revenue, though the Intermodal and Transportation Solutions segment, which is the asset-heavy part of Hub Group, did see an increase in operating income. Revenue was $528 million, down from $561 million a year earlier. But operating income rose to $14.4 million from $13.6 million a year ago. Logistics revenue was $404 million, down from $459 million. Adjusted operating income for the segment was $23 million compared to $26 million a year earlier. Sequentially, results at Hub Group were mostly weaker but only by minor amounts. Operating revenue sequentially dropped 1%. Purchased transportation was down less than half a percentage point. But operating income was down 8%. And net income declined 7.1%. Total legacy headcount, which excludes acquisition employees, drivers and warehouse employees, declined 3% from prior year. More articles by John Kingston Averitt pay increase could be a sign of some acceleration in driver wages Sequential numbers at diversified trucking operator TFI International may mark a turnaround At C.H. Robinson, improved profitability, productivity and a lot fewer workers The post Hub Group is fully behind a potential UP-NS transcontinental railroad creation appeared first on FreightWaves. Error in retrieving data Sign in to access your portfolio Error in retrieving data Error in retrieving data Error in retrieving data Error in retrieving data
Yahoo
7 minutes ago
- Yahoo
It's Liberation Day III - and from today, dozens of countries will face painful Trump tariffs
It is "Liberation Day" III - the third tariff deadline set by Donald Trump. From today, countries without bilateral trade agreements face reciprocal tariffs - ranging from 25% to 50% - with a baseline of 15% to 20% for any not making a deal. He has delayed twice, from April to July and from July to August, but hammered this date home in his trademark caps-on style: "THE AUGUST FIRST DEADLINE STANDS STRONG, AND WILL NOT BE EXTENDED. A BIG DAY FOR AMERICA!!!" "Will not be extended" for anyone but Mexico, it seems. The country secured a 90-day extension at the last minute, with Mr Trump citing the "complexities" of the border. By close of business on the eve of deadline, he had a handful of framework deals - some significant - including the UK (10%), the EU, Japan and South Korea (15%), Indonesia and the Philippines (19%), Vietnam (20%). On the EU agreement, which he struck in Scotland, the president said: "It's a very powerful deal, it's a big deal, it's the biggest of all the deals." But what happened to the "90 deals in 90 days" touted by the White House earlier this year? The short answer is they were replaced by letters of instruction to pay a tariff set by the US. Amid of flurry of late activity, the US played hardball with major trading partners like Canada. "For the rest of the world, we're going to have things done by Friday," said US Commerce Secretary Howard Lutnick - the "rest of the world" meaning everyone but China. There is, apparently, the "framework of a deal" between the world's two largest economies, but talks between Washington and Beijing are continuing. Read more US news: In terms of wins, he can claim some significant deals and point to his tariffs having generated an impressive $27bn (£20.4bn) in June, not bad for a single month. But the legality of the approach is under siege - with the US Court of International Trade ruling that the "Liberation Day" tariffs exceeded the president's authority, with enforcement paused pending appeal. The deadline has stirred the pot, forcing a handful of deals onto the table. Whether they stick or survive legal scrutiny is far from settled. But the playbook remains the same - threaten the world with trade chaos, whittle it down, celebrate the wins, and pray no one checks what's legal.
