‘This is coming for everyone': A new kind of AI bot takes over the web
To offer users a tidy AI summary instead of Google's '10 blue links,' companies such as OpenAI and Anthropic have started sending out bots to retrieve and recap content in real time. They are scraping webpages and loading relevant content into the AI's memory and 'reading' far more content than a human ever would.
Subscribe to The Post Most newsletter for the most important and interesting stories from The Washington Post.
According to data shared exclusively with The Washington Post, traffic from retrieval bots grew 49 percent in the first quarter of 2025 from the fourth quarter of 2024. The data is from TollBit, a New York-based start-up that helps news publishers monitor and make money when AI companies use their content.
TollBit's report, based on data from 266 websites - half of which are run by national and local news organizations - suggests that the growth of bots that retrieve information when a user prompts an AI model is on an exponential curve.
'It starts with publishers, but this is coming for everyone,' Toshit Panigrahi, CEO and co-founder of TollBit, said in an interview.
Panigrahi said that this kind of bot traffic, which can be hard for websites to detect, reflects growing demand for content, even as AI tools devastate traffic to news sites and other online platforms. 'Human eyeballs to your site decreased. But the net amount of content access, we believe, fundamentally is going to explode,' he said.
A spokesperson for OpenAI said that referral traffic to publishers from ChatGPT searches may be lower in quantity but that it reflects a stronger user intent compared with casual web browsing.
To capitalize on this shift, websites will need to reorient themselves to AI visitors rather than human ones, Panigrahi said. But he also acknowledged that squeezing payment for content when AI companies argue that scraping online data is fair use will be an uphill climb, especially as leading players make their newest AI visitors even harder to identify.
Debate around the AI industry's use of online content has centered on the gargantuan amounts of text needed to train the AI models that power tools like ChatGPT. To obtain that data, tech companies use bots that scrape the open web for free, which has led to a raft of lawsuits alleging copyright theft from book authors and media companies, including a New York Times lawsuit against OpenAI. Other news publishers have opted for licensing deals. (In April, The Washington Post inked a deal with OpenAI.)
In the past eight months, as chatbots have evolved to incorporate features like web search and 'reasoning' to answer more complex queries, traffic for retrieval bots has skyrocketed. It grew 2.5 times as fast as traffic for bots that scrape data for training between the fourth quarter of 2024 and the first quarter of 2025, according to TollBit's report.
Panigrahi said TollBit's data may underestimate the magnitude of this change because it doesn't reflect bots that AI companies send out on behalf of AI 'agents' that can complete tasks on a user's behalf, like ordering takeout from DoorDash.
The start-up's findings also add a new dimension to mounting evidence that the modern internet - optimized for Google search results and social media algorithms - will have to be restructured as the popularity of AI answers grows.
'To think of it as, 'Well, I'm optimizing my search for humans' is missing out on a big opportunity,' he said.
Installing TollBit's analytics platform is free for news publishers, and the company has more than 2,000 clients, many of which are struggling with these seismic changes, according to data in the report.
Although news publishers and other websites can implement blockers to prevent various AI bots from scraping their content, TollBit found that more than 26 million AI scrapes bypassed those blockers in March alone. Some AI companies claim bots for AI agents don't need to follow bot instructions because they are acting on behalf of a user.
Mark Howard, chief operating officer for the media company Time, a TollBit client, said the start-up's traffic data has helped Time negotiate content licensing deals with AI companies including OpenAI and the search engine Perplexity.
But the market to fairly compensate publishers is far from established, Howard said. 'The vast majority of the AI bots out there absolutely are not sourcing the content through any kind of paid mechanism. … There is a very, very long way to go.'
Related Content
He's dying. She's pregnant. His one last wish is to fight his cancer long enough to see his baby.
The U.S. granted these journalists asylum. Then it fired them.
'Enough is enough.' Why Los Angeles is still protesting, despite fear.
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Yahoo
15 minutes ago
- Yahoo
If You'd Invested $1,000 in Solana 5 Years Ago, Here's How Much You'd Have Today
Key Points Solana runs on a proof-of-stake network that is one of the fastest in the crypto world. The network is already processing thousands of transactions per second. The technical strength of the network has made it a home run for investors. 10 stocks we like better than Solana › Only launched about 5.5 years ago, Solana (CRYPTO: SOL) is now the sixth-largest cryptocurrency in the world with a market cap of over $96 billion as of July 30. Many investors see immense potential in Solana's network. It's one of the few cryptocurrencies to operate on a proof-of-stake (PoS) mechanism to govern the network. After realizing how energy-intensive the traditional crypto-mining, proof-of-work (PoW) system had become on Bitcoin, the world's largest cryptocurrency, several crypto networks transitioned to PoS. Instead of using high computing power to solve a puzzle like with PoW, PoS has investors stake their tokens to the network, and then assigns them at random to validate transactions and mint new tokens. The more tokens one stakes, the higher the chance they have of being selected and also earning rewards. Even more unique, Solana's network also has a proof-of-history mechanism that essentially creates a sequential record of transactions, enabling even faster transactions on the network. As a result, Solana's network can process thousands of transactions per second (TPS), but it has the theoretical potential to process up to 65,000 TPS, if not more. This gives Solana and its network immense potential to disrupt the global payments system. Investors have done well While volatile like most cryptocurrencies, Solana has been a huge winner for investors that bought the token five years ago. The technical strength of its network has made Solana one of the few altcoins that investors see a strong use case for. Roughly five years ago, Solana traded for just $1.73. Today, it trades for over $179. That's a gain of roughly 10,264%. So, if you invested $1,000 in Solana five years ago, you now have $103,636! That's simply incredible. Investors aren't likely to find too many investments like that in their lifetime. Should you invest $1,000 in Solana right now? Before you buy stock in Solana, consider this: The Motley Fool Stock Advisor analyst team just identified what they believe are the for investors to buy now… and Solana wasn't one of them. The 10 stocks that made the cut could produce monster returns in the coming years. Consider when Netflix made this list on December 17, 2004... if you invested $1,000 at the time of our recommendation, you'd have $624,823!* Or when Nvidia made this list on April 15, 2005... if you invested $1,000 at the time of our recommendation, you'd have $1,064,820!* Now, it's worth noting Stock Advisor's total average return is 1,019% — a market-crushing outperformance compared to 178% for the S&P 500. Don't miss out on the latest top 10 list, available when you join Stock Advisor. See the 10 stocks » *Stock Advisor returns as of July 29, 2025 Bram Berkowitz has positions in Bitcoin. The Motley Fool has positions in and recommends Bitcoin and Solana. The Motley Fool has a disclosure policy. If You'd Invested $1,000 in Solana 5 Years Ago, Here's How Much You'd Have Today was originally published by The Motley Fool
Washington Post
17 minutes ago
- Washington Post
Cowboys owner Jerry Jones dismisses Micah Parsons' trade request as negotiation talk
OXNARD, Calif. — Dallas Cowboys owner and general manager Jerry Jones does not intend to trade Micah Parsons after the star defensive end said he wants to leave the team amid a breakdown in negotiating a contract extension. 'Surely you guys have been around this stuff and know how to recognize negotiation talk, that type of thing. And so that's where I put that,' Jones said after practice Saturday, one day after Parsons posted on social media to request a trade .
Forbes
17 minutes ago
- Forbes
You Want To Speak Memorably? Try These Smart Tips
In today's high-stakes professional world, the ability to communicate isn't just a competency—it's currency. Are you leading a team? Strong communication skill is one of your most important tools. Pitching a product? Ditto. Navigating a job interview? Ditto. In most every workplace situation, your presentation style can be the defining factor between influence and invisibility. Bill McGowan understands this better than most. This Emmy Award-winning TV journalist and bestselling author is a top communications advisor to leading names in business, tech, entertainment, sports, and finance. He's coached the founders of Amazon, Meta, Twitter, LinkedIn, Instagram, Spotify, Snapchat, and Airbnb … as well as Oscar, Emmy, and Grammy winners and World Series, Super Bowl, and Olympic champions. McGowan's latest book is Speak, Memorably: The Art of Captivating an Audience. His experience in TV journalism underscored many of the lessons he now provides as a coach. For example, when McGowan worked on ABC's popular '20/20' show, the executive producer would share the minute-to-minute ratings of a segment. Any time an on-camera personality told a story, viewers' interest spiked. Stories, McGowan learned, are much more compelling than statistics. You want to break out of the conformity zone? McGowan recommends avoiding 'corporate speak' and communicating concisely with clarity and simplicity. And at every opportunity, use pertinent stories. While it's important that every part of a presentation be interesting and relevant, McGowan highlights what psychologists call the primary-recency effect: people tend to remember the first and last items in a sequence better than those in the middle. That's a clue on where compelling stories, challenges, or facts should be in a presentation. To underscore the importance and power of simplicity, McGowan tells of a CEO who was in a media interview about a struggle his company was experiencing. The interviewer asked the CEO, 'Why do you think your IPO underperformed vis-à-vis the other companies in your competitive space?' McGowan says that instead of blaming the market or a poor valuation or even suggesting that they didn't have the right financial institution leading the offering, the CEO simply said, 'We underperformed because we failed to come up with a narrative that could fit on the back of a cocktail napkin.' McGowan used that story to kick off a speech he made to a group of communication clients. Using a 'through line,' the connective tissue that helps explain the relevance of a story, he spoke on the subject of simplicity. Citing research studies, McGowan says factual information embedded within storytelling becomes 22 times more memorable 'than if you just fire hose talking points and data points.' McGowan coaches his clients to communicate the visual, the specific, and the anecdotal. When you speak abstractly and theoretically, it's pictureless, he says. 'We human beings have this movie reel spinning in our heads at all times. If a speaker doesn't give us imagery to work with, that movie reel keeps spinning in our head—only now it's spinning with the images we've created. That's what daydreaming is.' In what ways has the rise of remote work and virtual meetings affected the way people communicate? In addition to what many people now call 'Zoom fatigue,' McGowan talks about 'Zoomnesia.' That occurs when people attend a seemingly endless string of remote meetings in which they're sitting in the same room with the same potted plant on their desk and the same view out the window and with no contextual clues. That circumstance makes it even harder for people to retain information, and it underscores the value of well thought-out communication practices that engage people's thinking. McGowan frequently uses metaphors in his coaching. For example, he talks about what he calls the 'pasta sauce principle.' You put a pot of tomato sauce on the stove and cook it down for about three hours. You then have something with about half the volume but a lot more flavor. 'But it takes time and effort to get that reduction going,' McGowan says. 'The same is true for speaking.' It wouldn't be accurate to say McGowan has written a cookbook. But Speak, Memorably certainly provides the ingredients for welcome servings of appetizing communication. Whether you're giving a TED talk, speaking at the local Rotary Club, or just trying not to lose people in your Monday morning staff meeting, McGowan offers insight that will boost your communication mojo.
