
The internet of agents is rising fast, and publishers are nowhere near ready
Now imagine one day a robot comes in to buy books on behalf of someone. It ignores the displays, the coffee kiosk, and the tchotchkes near the till. It just grabs the book the person ordered, pays for it, and walks out. The next day 4 robots come in, then 12 the day after that. Soon, robots are outnumbering humans in your store, which are dwindling by the day. You soon see very few sales from nonbook items, publishers stop bothering with those displays, and the coffee goes cold. Revenue plummets.
In response, you might start charging robots a fee to enter your store, and if they don't pay it, you deny them entry. But then one day a robot that looks just like a human comes in—to the point that you can't tell the difference. What do you do then?
This analogy is basically what the publishing world is going through right now, with bot traffic to media websites skyrocketing over the past three months. That's according to new data from TollBit, which recently published its State of the Bots report for the first quarter of 2025. Even more concerning, however, is that the most popular AI search engines are choosing to ignore long-respected standards for blocking bots, in some cases arguing that when a search 'agent' acts on behalf of an individual user, the bot should be treated as human.
The robot revolution
TollBit's report paints a fast-changing picture of what's happening with AI search. Over the past several months, AI companies have either introduced search abilities or greatly increased their search activity. Bot scraping focused on retrieval-augmented generation (RAG), which is distinct from training data, increased 49% over the previous quarters. Anthropic's Claude notably introduced search, and in the same period ChatGPT (the world's most popular chatbot by far) had a spike in users, plus deep research tools from all the major providers began to take hold.
At the same time, publishers increased their defenses. The report reveals that media websites in January were using various methods to block AI bots four times as much as they were doing in a year before. The first line of defense is to adjust their website's robots.txt file, which tells which specific bots are welcome and which ones are forbidden from accessing the content.
The thing is, adhering to robots.txt is ultimately an honor system and not really enforceable. And the report indicates more AI companies are treating it as such: Among sites in TollBit's network, bot scrapes that ignore robots.txt increased from 3.3% to 12.9% in just one quarter.
Part of that increase is due to a relatively new stance the AI companies have taken, and it's subtle but important. Broadly speaking, there are three different kinds of bots that scrape or crawl content:
Training bots: These are bots that crawl the internet to scrape content to provide training data for AI models.
Search indexing bots: Bots that go out and crawl the web to ensure the model has fast access to important information outside its training set (which is usually out of date). This is a form of RAG.
User agent bots: Also a form of RAG, these are crawlers that go out to the web in real time to find information directly in response to a user query, regardless of whether the content it finds has been previously indexed.
Because No. 3 is an agent acting on behalf of a human, AI companies argue that it's an extension of that user behavior and have essentially given themselves permission to ignore robots.txt settings for that use case. This isn't guesswork— Google, Meta, and Perplexity have made it explicit in their developer notes. This is how you get human-looking robots in the bookstore.
When humans go to websites, they see ads. Humans can be intrigued or enticed by other content, such as a link to a podcast about the same topic as an article they're reading. Humans can decide whether or not to pay for a subscription. Humans sometimes choose to make a transaction based on the information in front of them.
Bots don't really do any of that (not yet, anyway). Large parts of the internet economy depend on human attention to websites, but as the report shows, that behavior drops off massively when someone uses AI to search the web—AI search engines provide very little in the way of referral traffic compared to traditional search. This of course is what's behind many of the lawsuits now in play between media companies and AI companies. How that is resolved in the legal realm is still TBD, but in the meantime, some media sites are choosing to block bots—or at least are attempting to—from accessing their content at all.
For user agent bots, however, that ability has been taken away. The AI companies have always seen data harvesting in the way that's most favorable to their insatiable demand for it, famously claiming that data only needs to be 'publicly available' to qualify as training data. Even when they claim to respect robots.txt for their search engines, it's an open secret that they sometimes use third-party scrapers to bypass it.
Unmasking the bots
So apart from suing and hoping for the best, how can publishers regain some, well, agency in the emerging world of agent traffic? If you believe AI substitution threatens your growth, there are additional defenses to consider. Hard paywalls are easier to defend, both technically and legally, and there are several companies (including TollBit, but there are others, such as ScalePost) that specialize in redirecting bot traffic to paywalled endpoints specifically for bots. If the robot doesn't pay, it's denied the content, at least in theory.
Collective action is another possibility. I doubt publishers would launch a class action around this specific relabeling of user agents, but it does provide more ammunition in broader copyright lawsuits. Besides going to court, industry associations could come out against the move. The News/Media Alliance in particular has been very vocal about AI companies' alleged transgressions of copyright.
The idea of treating agentic activity as the equivalent of human activity has consequences that go beyond the media. Any content or tool that's been traditionally available for free will need to reevaluate that access now that robots are destined to be a growing part of the mix. If there was any doubt that simply updating robots.txt instructions was adequate, the TollBit report blew it out of the water.
The stance that 'AI is just doing what humans do' is often used as a defense for when AI systems ingest large amounts of information and then produce new content based on it. Now the makers of those systems are quietly extending that idea, allowing their agents to effectively impersonate humans while shopping the web for data. Until it's clear how to build profitable stores for robots, there should be a way to force their masks off.

Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Yahoo
23 minutes ago
- Yahoo
Regulations can hurt businesses. Federal government met with 30+ in Knoxville to learn how
On the back roads of Hardin Valley, an automated robot being designed for the Department of Defense sits tucked away in a warehouse. Navus is a family-run automation solutions company with only 25 employees, yet it finds ways to secure national-level contracts. Sam Lawless, the business development manager, led two federal representatives through the back of the warehouse to explain what his father's company is all about. Two representatives from the U.S. Small Business Administration, Michael Vallante and Tyler Teresa, met with more than 30 Knoxville business owners at a round-table discussion about the roadblocks and pitfalls from federal regulations that can hinder small businesses. "I hate saying, 'I'm from the government, and I'm here to help', but I really am," Teresa, the SBA regional administrator, told Knox News. Teresa and Vallante toured both Navus and BioPet Laboratories, a biotechnology company specializing in canine DNA, to get a firsthand look at some of the struggles small businesses are facing. Lawless' father started Navus in 1998 as a full-blown automation manufacturing company. The company has smaller, local contracts, but it's the contract with the Department of Defense that has been keeping Navus in business since 2007. Lawless turned to show Teresa and Vallante the robot, which streamlines welding for U.S. Navy submarine missiles. Before Navus' work automated the process, welders would be lowered 40 feet into a metal pipe being heated to 350 degrees. Each welder wore a heat-resistant suit and worked for as long as they could handle − only about five minutes − before switching to another worker, Lawless said. Navus effectively shortened a dangerous, complicated process that took about eight months down to just three. Funding massive projects is difficult, Lawless said, but working with people at the SBA will hopefully free up money to help keep Navus in business. "I felt like we've kind of been suffering in silence," Lawless said. "I left (the round table) really happy because I felt like the stuff we're dealing with, we're not dealing with it alone. A lot of other small businesses are dealing with it." Teresa said the SBA exists to be an advocate for businesses like Navus and BioPet. The roadblocks businesses hit aren't always seen in Washington, D.C., and he hopes to help bridge the gap. "We've got to make it easier on businesses. Some federal regulations are harming small businesses, and that's because someone in D.C. created them, and they're not here talking to people," Teresa said. "So, that's why it's important to actually hear from these businesses directly to see what they're dealing with and what issues need to be lifted to help them." J Retinger is the CEO of BioPet Laboratories, which specializes in their signature "PooPrints" that determine which dog's owner is being irresponsible. If someone in a residential complex doesn't pick up after their pet, sending a sample to Retinger's company can help solve the mystery. BioPet, which has 49 employees, runs into federal roadblocks when dealing with customs and trying to secure loans, he said. Figuring out a way around can feel a lot like talking to a wall, but discussions with SBA representatives help them see a path forward. "It's the first step in the relationship, and now having a name and a face who have seen what we do, I think it's a huge step," Retinger said. "It's the first time we've had federal people visit." Vallante said there's a fine line between success and failure for entrepreneurs, and the SBA wants to help make sure the government tips the scales toward success. "Our job is to talk to small businesses, find out what's going on, and bring that back up to Washington," he said. Knox News reporter Joanna Hayes covers restaurants and retail for the business growth and development team. Email: Instagram: @knoxeat65. Sign up for Joanna's Eat65 email newsletter to get the latest drink and dining news, as well as restaurant recommendations at Support strong local journalism by subscribing at This article originally appeared on Knoxville News Sentinel: Small Business Administration in Knoxville to talk federal regulations Solve the daily Crossword
Yahoo
23 minutes ago
- Yahoo
Magnificent 7 Diverge as Peak Earnings Season Puts Spotlight on Big Tech
Peak earnings season for the second quarter begins this week and lasts through the second week of August. This is when roughly 75% of S&P 500 companies are scheduled to report. Thus far 34% of companies from the index have reported, and the early results bode well for the rest of the season. Last week investors focused on results from the first two Magnificent 7 constituents, Tesla and Alphabet, which both reported on Wednesday and had differing results. Warning! GuruFocus has detected 7 Warning Signs with REG. Not surprisingly, Tesla continued to report waning demand. This was expected after the EV maker released disappointing results in their Second Quarter 2025 Production, Deliveries & Deployments on July 2. That report showed that second quarter vehicle deliveries declined 14%, the second straight quarterly decline.[1] In their second quarter earnings report, Tesla missed expectations on the top and bottom-line due to automotive revenue which was down 16% YoY, in part due to competition from Chinese EV makers which are producing more affordable models.[2] The stock is down ~16% YTD. On the flipside, Google parent, Alphabet, was a bright spot. Alphabet beat expectations on the top and bottom-line, driven by robust revenue from YouTube advertising and Google Cloud. The company also offered 2025 CapEx estimates that were higher than Wall Street's expectations for the second time this year. In February, Alphabet said it expected to spend $75B in capital expenditures (CAPEX) mainly due to the expansion in its AI capabilities, which was higher than the $58.84B analysts had expected. Last week that number rose to $85B, this time due to strong and growing demand for our Cloud products and services. Alphabet CFO, Anat Ashkenazi, told investors to expect further increases to CAPEX in 2026.[3] Alphabet stock is up ~2% YTD. With the results from those names as well as the 110 other S&P 500 constituents that reported last week, Q2 growth propelled to 6.4% from 5.6% the prior week. Revenues also increased to 5.1% from 4.4% the week prior. Beat rates remain impressive with 80% of S&P 500 companies surpassing expectations on the top and bottom-line, better than the 1, 5, and 10-year beat rate averages according to FactSet.[4] On Deck this Week - More Big Tech Results This week all eyes will be on four more Magnificent 7 reports from Meta and Microsoft on Wednesday, and Apple and Amazon on Thursday. Nvidia is the final of the group to report on August 27. Source: Wall Street Horizon Outlier Earnings Dates This Week Academic research shows that, when a company confirms a quarterly earnings date that is later than when they have historically reported, it's typically a sign that the company will share negative news on their upcoming call, while moving a release date earlier suggests the opposite.[5] This week we get results from a number of large companies on major indexes that have pushed their Q2 2025 earnings dates outside of their historical norms. Eight companies within the S&P 500 confirmed outlier earnings dates for this week, four of which are earlier than usual and therefore have positive DateBreaks Factors*. Those names are Regency Centers Corp (NASDAQ:REG), Booking Holdings (NASDAQ:BKNG), AES Corp (NYSE:AES) and CVS Health Corp (NYSE:CVS). The four companies with negative DateBreak Factors for this week are Teradyne Inc (NASDAQ:TER), PPG Industries (NYSE:PPG), UnitedHealth Group (NYSE:UNH), and Kimberly Clark Corp (NASDAQ:KMB). * Wall Street Horizon DateBreaks Factor: statistical measurement of how an earnings date (confirmed or revised) compares to the reporting company's 5-year trend for the same quarter. Negative means the earnings date is confirmed to be later than historical average while Positive is earlier. Q2 Earnings Wave The peak weeks of the Q2 earnings season are expected to fall between July 28 - August 15, with each week expected to see over 2,000 reports. Currently, August 7 is predicted to be the most active day with 1,291 companies anticipated to report. Thus far, 71% of companies have confirmed their earnings date, and 14% have reported (out of our universe of 11,000+ global names), so this is subject to change. The remaining dates are estimated based on historical reporting data. Source: Wall Street Horizon 1 Tesla Second Quarter 2025 Production, Deliveries & Deployments, July 2, 2025, Tesla Q2 2025 Quarterly Earnings Update, July 23, 2025, Alphabet Announces Second Quarter 2025 Results, July 23, 2025, FactSet Earnings Insight, John Butters, July 25, 2025, Time Will Tell: Information in the Timing of Scheduled Earnings News, Journal of Financial and Quantitative Analysis, Eric C. So, Travis L. Johnson, Dec, 2018, Copyright 2025 Wall Street Horizon, Inc. All rights reserved. Do not copy, distribute, sell or modify this document without Wall Street Horizon's prior written consent. This information is provided for information purposes only. Neither TMX Group Limited nor any of its affiliated companies guarantees the completeness of the information contained in this publication, and we are not responsible for any errors or omissions in or your use of, or reliance on, the information. This publication is not intended to provide legal, accounting, tax, investment, financial or other advice and should not be relied upon for such advice. The information provided is not an invitation to purchase securities, including any listed on Toronto Stock Exchange and/or TSX Venture Exchange. TMX Group and its affiliated companies do not endorse or recommend any securities referenced in this publication. This publication shall not constitute an offer to sell or the solicitation of an offer to buy, nor may there be any sale of any securities in any state or jurisdiction in which such offer, solicitation or sale would be unlawful prior to registration or qualification under the securities laws of any such state or jurisdiction. TMX, the TMX design, TMX Group, Toronto Stock Exchange, TSX, and TSX Venture Exchange are the trademarks of TSX Inc. and are used under license. Wall Street Horizon is the trademark of Wall Street Horizon, Inc. All other trademarks used in this publication are the property of their respective owners. This article first appeared on GuruFocus. Error in retrieving data Sign in to access your portfolio Error in retrieving data Error in retrieving data Error in retrieving data Error in retrieving data
Yahoo
23 minutes ago
- Yahoo
Asheville cocktail bar named one of best in US. Your guide to its menu, lively events
ASHEVILLE - Little Jumbo, a laid-back yet high-spirited cocktail bar where locals go to relax and play, has been a staple in the historic Montford and Five Points neighborhoods for eight years. The building was constructed in the Roaring 20s for a grocery store and has been preserved to serve the community with intricately crafted drinks and a good time any time of the week. On July 30, USA TODAY named Little Jumbo one of the 29 USA TODAY Bars of the Year for 2025. 'It's incredibly flattering,' said Chall Gray, who literally wrote the book on cocktail bars in 'The Cocktail Bar: Notes for an Owner & Operator.' He and Little Jumbo co-owner Jay Sanders also partner with the bar concept design and cocktail program revamp company, Slings and Arrows. More: Best Bars in America: How USA TODAY picked the 2025 finalists 'It's cool to see that the unique and offbeat way that we do things, and all of our peculiarities, resonate with people," Gray said. What makes Little Jumbo in Asheville, North Carolina stand out Little Jumbo maintains its low-key local feel on the cusp of two historic neighborhoods, less than a mile from downtown Asheville. In the tourist-driven mountain city, it's a hidden gem for visitors to discover and a jewel of the city for residents. Formerly the J.H. Jenkins grocery store in the 1920s, the storefront has transitioned over the decades to become a go-to destination for quality handcrafted cocktails and an unforgettable night, where the theme and experience change depending on the night. Dubbed the 'living room of Montford,' husband and wife Chall and Lucia Gray, along with Jay Sanders, opened Little Jumbo to be reminiscent of a 19th-century Irish pub, meets Western saloon, meets Victorian parlor. Little Jumbo's original structural bones, like its tin ceiling panels, exposed brick walls and wooden beams, set the backdrop. The design additions of vintage furniture, wallpaper, framed photos and lighting fixtures create a warmth and intimacy of a Victorian-style parlor. More: USA TODAY named an Asheville bar one of America's best. We wish these spots made the cut It presents a timeless scene where bourbon flows, Old Fashioneds reign, and Manhattan silver platter table service for two coexist with modern concoctions and classic cocktails with a twist. Contemporary, rotating art installations by local artist-in-residence Wade Asa's abstract creature hint at Little Jumbo's flair for quirk. The annual February "Tropilachia Club" bar flip transforms Little Jumbo into a tiki bar-style cocktail oasis, dispelling notions of it being a pretentious bar and instead affirming the visionaries' adeptness in creating a space that can transform from a lively neighborhood bar to a serene tropical escape. Sanders developed Ophelia's Cocktail Companion software, which stores nearly 1,000 cocktail recipes that the staff can study and use to make on- and off-menu cocktails on any given day. Monthly, on Saturdays, themed deejayed dance parties, such as the Halloween in July costume party, kick up the energy. It's juxtaposed by the Sunday afternoon "Authors in Conversation" series, a chat featuring literary artists. The Monday live jazz nights pack out the bar as local musicians take turns on the makeshift stage area. The popularity of the event series inspired the addition of Tuesday jazz nights, where Sanders, a seasoned musician, leads a four-piece jazz quartet weekly. Or stop in for a rousing game night on Wednesdays. Yahtzee or Backgammon, anyone? What to order at Little Jumbo Little Jumbo's booklet menu features cocktails, mocktails, beer, cider, wine and spirit neat-pours. Unsure about a cocktail ingredient or style? A glossary, illustrations of glassware and ice types, and light-hearted descriptions, along with the well-versed bartenders, will guide you through. Prefer not to imbibe in alcohol? Little Jumbo puts the same devotion to craft cocktail making into its mocktail line-up. Old Fashioned. The father of all cocktails gets a refreshing twist at Little Jumbo. Order it as a classic, or journey through the ageless varieties, like gin, 1870s and 1930s Old Fashioned. Hello Friday. Inspired by the Angostura Highball, Hello Friday is a low-ABV cocktail, and an original menu mainstay made with bonded bourbon, Angostura bitters and Maine root spicy ginger brew. It can be altered as a nonalcoholic Hello Monday or a boozier version, The Rodriguez. More: USA TODAY named its Bars of the Year. This Asheville favorite made the list Water Lily. Keep it classy with the petite and powerful Water Lily, made with gin, crème de violette, orange curaçao and lemon served in a coupe glass. Tequila spritz. The featured tequila spritz changes with the season. This summer, the Bandito ― a cross between an Aperol spritz and tequila sunrise ― is making a big splash with imbibers. It's made with blanco tequila, Bizarro aperitivo, grapefruit, orange, pineapple, lime, peach bitters, cava, soda and with Tajin Chamoy rim. A refreshing choice for lounging on the side patio. Insider tip: Love Little Jumbo? The cocktail bar connoisseurs behind the North Asheville bar opened a second watering hole last fall in West Asheville. Character Study, a literary-themed, bright, airy cocktail bar, allows patrons to add displayed books to their bar tab, offers cocktail-of-the-day specials, and is outfitted with carrels ― intimate library-style cubicle desks ― ideal for private conversations and study nights. Details: 241 Broadway St., Asheville; Instagram: @littlejumbobar. Tiana Kennell is the food and dining reporter for the Asheville Citizen Times, part of the USA Today Network. Tips, comments, questions? Email tkennell@ or follow @PrincessOfPage on Instagram/Bluesky. This article originally appeared on Asheville Citizen Times: Asheville neighborhood bar named one of best by USA TODAY Solve the daily Crossword