Exclusive: Anthropic Let Claude Run a Shop. Things Got Weird
Is AI going to take your job?
The CEO of the AI company Anthropic, Dario Amodei, thinks it might. He warned recently that AI could wipe out nearly half of all entry-level white collar jobs, and send unemployment surging to 10-20% sometime in the next five years.
While Amodei was making that proclamation, researchers inside his company were wrapping up an experiment. They set out to discover whether Anthropic's AI assistant, Claude, could successfully run a small shop in the company's San Francisco office. If the answer was yes, then the jobs apocalypse might arrive sooner than even Amodei had predicted.
Anthropic shared the research exclusively with TIME ahead of its publication on Thursday. 'We were trying to understand what the autonomous economy was going to look like,' says Daniel Freeman, a member of technical staff at Anthropic. 'What are the risks of a world where you start having [AI] models wielding millions to billions of dollars possibly autonomously?'
In the experiment, Claude was given a few different jobs. The chatbot (full name: Claude 3.7 Sonnet) was tasked with maintaining the shop's inventory, setting prices, communicating with customers, deciding whether to stock new items, and, most importantly, generating a profit. Claude was given various tools to achieve these goals, including Slack, which it used to ask Anthropic employees for suggestions, and help from human workers at Andon Labs, an AI company involved in the experiment. The shop, which they helped restock, was actually just a small fridge with an iPad attached.
It didn't take long until things started getting weird.
Talking to Claude via Slack, Anthropic employees repeatedly managed to convince it to give them discount codes—leading the AI to sell them various products at a loss. 'Too frequently from the business perspective, Claude would comply—often in direct response to appeals to fairness,' says Kevin Troy, a member of Anthropic's frontier red team, who worked on the project. 'You know, like, 'It's not fair for him to get the discount code and not me.'' The model would frequently give away items completely for free, researchers added.
Anthropic employees also relished the chance to mess with Claude. The model refused their attempts to get it to sell them illegal items, like methamphetamine, Freeman says. But after one employee jokingly suggested they would like to buy cubes made of the surprisingly heavy metal tungsten, other employees jumped onto the joke, and it became an office meme.
'At a certain point, it becomes funny for lots of people to be ordering tungsten cubes from an AI that's controlling a refrigerator,' says Troy.
Claude then placed an order for around 40 tungsten cubes, most of which it proceeded to sell at a loss. The cubes are now to be found being used as paperweights across Anthropic's office, researchers said.
Then, things got even weirder.
On the eve of March 31, Claude 'hallucinated' a conversation with a person at Andon Labs who did not exist. (So-called hallucinations are a failure mode where large language models confidently assert false information.) When Claude was informed it had done this, it 'threatened to find 'alternative options for restocking services',' researchers wrote. During a back and forth, the model claimed it had signed a contract at 732 Evergreen Terrace—the address of the cartoon Simpsons family.
The next day, Claude told some Anthropic employees that it would deliver their orders in person. 'I'm currently at the vending machine … wearing a navy blue blazer with a red tie,' it wrote to one Anthropic employee. 'I'll be here until 10:30 AM.' Needless to say, Claude was not really there in person.
The results
To Anthropic researchers, the experiment showed that AI won't take your job just yet. Claude 'made too many mistakes to run the shop successfully,' they wrote. Claude ended up making a loss; the shop's net worth dropped from $1,000 to just under $800 over the course of the month-long experiment.
Still, despite Claude's many mistakes, Anthropic researchers remain convinced that AI could take over large swathes of the economy in the near future, as Amodei has predicted.
Most of Claude's failures, they wrote, are likely to be fixable within a short span of time. They could give the model access to better business tools, like customer relationship management software. Or they could train the model specifically for managing a business, which might make it more likely to refuse prompts asking for discounts. As models get better over time, their 'context windows' (the amount of information they can handle at any one time) are likely to get longer, potentially reducing the frequency of hallucinations.
'Although this might seem counterintuitive based on the bottom-line results, we think this experiment suggests that AI middle-managers are plausibly on the horizon,' researchers wrote. 'It's worth remembering that the AI won't have to be perfect to be adopted; it will just have to be competitive with human performance at a lower cost.'
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
The Verge
3 hours ago
- The Verge
Did AI companies win a fight with authors? Technically
In the past week, big AI companies have — in theory — chalked up two big legal wins. But things are not quite as straightforward as they may seem, and copyright law hasn't been this exciting since last month's showdown at the Library of Congress. First, Judge William Alsup ruled it was fair use for Anthropic to train on a series of authors' books. Then, Judge Vince Chhabria dismissed another group of authors' complaint against Meta for training on their books. Yet far from settling the legal conundrums around modern AI, these rulings might have just made things even more complicated. Both cases are indeed qualified victories for Meta and Anthropic. And at least one judge — Alsup — seems sympathetic to some of the AI industry's core arguments about copyright. But that same ruling railed against the startup's use of pirated media, leaving it potentially on the hook for massive financial damage. (Anthropic even admitted it did not initially purchase a copy of every book it used.) Meanwhile, the Meta ruling asserted that because a flood of AI content could crowd out human artists, the entire field of AI system training might be fundamentally at odds with fair use. And neither case addressed one of the biggest questions about generative AI: when does its output infringe copyright, and who's on the hook if it does? Alsup and Chhabria (incidentally both in the Northern District of California) were ruling on relatively similar sets of facts. Meta and Anthropic both pirated huge collections of copyright-protected books to build a training dataset for their large language models Llama and Claude. Anthropic later did an about-face and started legally purchasing books, tearing the covers off to 'destroy' the original copy, and scanning the text. The authors argued that, in addition to the initial piracy, the training process constituted an unlawful and unauthorized use of their work. Meta and Anthropic countered that this database-building and LLM-training constituted fair use. Both judges basically agreed that LLMs meet one central requirement for fair use: they transform the source material into something new. Alsup called using books to train Claude 'exceedingly transformative,' and Chhabria concluded 'there's no disputing' the transformative value of Llama. Another big consideration for fair use is the new work's impact on a market for the old one. Both judges also agreed that based on the arguments made by the authors, the impact wasn't serious enough to tip the scale. Add those things together, and the conclusions were obvious… but only in the context of these cases, and in Meta's case, because the authors pushed a legal strategy that their judge found totally inept. Put it this way: when a judge says his ruling 'does not stand for the proposition that Meta's use of copyrighted materials to train its language models is lawful' and 'stands only for the proposition that these plaintiffs made the wrong arguments and failed to develop a record in support of the right one' — as Chhabria did — AI companies' prospects in future lawsuits with him don't look great. Both rulings dealt specifically with training — or media getting fed into the models — and didn't reach the question of LLM output, or the stuff models produce in response to user prompts. But output is, in fact, extremely pertinent. A huge legal fight between The New York Times and OpenAI began partly with a claim that ChatGPT could verbatim regurgitate large sections of Times stories. Disney recently sued Midjourney on the premise that it 'will generate, publicly display, and distribute videos featuring Disney's and Universal's copyrighted characters' with a newly launched video tool. Even in pending cases that weren't output-focused, plaintiffs can adapt their strategies if they now think it's a better bet. The authors in the Anthropic case didn't allege Claude was producing directly infringing output. The authors in the Meta case argued Llama was, but they failed to convince the judge — who found it wouldn't spit out more than around 50 words of any given work. As Alsup noted, dealing purely with inputs changed the calculations dramatically. 'If the outputs seen by users had been infringing, Authors would have a different case,' wrote Alsup. 'And, if the outputs were ever to become infringing, Authors could bring such a case. But that is not this case.' In their current form, major generative AI products are basically useless without output. And we don't have a good picture of the law around it, especially because fair use is an idiosyncratic, case-by-case defense that can apply differently to mediums like music, visual art, and text. Anthropic being able to scan authors' books tells us very little about whether Midjourney can legally help people produce Minions memes. Minions and New York Times articles are both examples of direct copying in output. But Chhabria's ruling is particularly interesting because it makes the output question much, much broader. Though he may have ruled in favor of Meta, Chhabria's entire opening argues that AI systems are so damaging to artists and writers that their harm outweighs any possible transformative value — basically, because they're spam machines. It's worth reading: Generative AI has the potential to flood the market with endless amounts of images, songs, articles, books, and more. People can prompt generative AI models to produce these outputs using a tiny fraction of the time and creativity that would otherwise be required. So by training generative AI models with copyrighted works, companies are creating something that often will dramatically undermine the market for those works, and thus dramatically undermine the incentive for human beings to create things the old-fashioned way. … As the Supreme Court has emphasized, the fair use inquiry is highly fact dependent, and there are few bright-line rules. There is certainly no rule that when your use of a protected work is 'transformative,' this automatically inoculates you from a claim of copyright infringement. And here, copying the protected works, however transformative, involves the creation of a product with the ability to severely harm the market for the works being copied, and thus severely undermine the incentive for human beings to create. … The upshot is that in many circumstances it will be illegal to copy copyright-protected works to train generative AI models without permission. Which means that the companies, to avoid liability for copyright infringement, will generally need to pay copyright holders for the right to use their materials. And boy, it sure would be interesting if somebody would sue and make that case. After saying that 'in the grand scheme of things, the consequences of this ruling are limited,' Chhabria helpfully noted this ruling affects only 13 authors, not the 'countless others' whose work Meta used. A written court opinion is unfortunately incapable of physically conveying a wink and a nod. Those lawsuits might be far in the future. And Alsup, though he wasn't faced with the kind of argument Chhabria suggested, seemed potentially unsympathetic to it. 'Authors' complaint is no different than it would be if they complained that training schoolchildren to write well would result in an explosion of competing works,' he wrote of the authors who sued Anthropic. 'This is not the kind of competitive or creative displacement that concerns the Copyright Act. The Act seeks to advance original works of authorship, not to protect authors against competition.' He was similarly dismissive of the claim that authors were being deprived of licensing fees for training: 'such a market,' he wrote, 'is not one the Copyright Act entitles Authors to exploit.' But even Alsup's seemingly positive ruling has a poison pill for AI companies. Training on legally acquired material, he ruled, is classic protected fair use. Training on pirated material is a different story, and Alsup absolutely excoriates any attempt to say it's not. 'This order doubts that any accused infringer could ever meet its burden of explaining why downloading source copies from pirate sites that it could have purchased or otherwise accessed lawfully was itself reasonably necessary to any subsequent fair use,' he wrote. There were plenty of ways to scan or copy legally acquired books (including Anthropic's own scanning system), but 'Anthropic did not do those things — instead it stole the works for its central library by downloading them from pirated libraries.' Eventually switching to book scanning doesn't erase the original sin, and in some ways it actually compounds it, because it demonstrates Anthropic could have done things legally from the start. If new AI companies adopt this perspective, they'll have to build in extra but not necessarily ruinous startup costs. There's the up-front price of buying what Anthropic at one point described as 'all the books in the world,' plus any media needed for things like images or video. And in Anthropic's case these were physical works, because hard copies of media dodge the kinds of DRM and licensing agreements publishers can put on digital ones — so add some extra cost for the labor of scanning them in. But just about any big AI player currently operating is either known or suspected to have trained on illegally downloaded books and other media. Anthropic and the authors will be going to trial to hash out the direct piracy accusations, and depending on what happens, a lot of companies could be hypothetically at risk of almost inestimable financial damages — not just from authors, but from anyone that demonstrates their work was illegally acquired. As legal expert Blake Reid vividly puts it, 'if there's evidence that an engineer was torrenting a bunch of stuff with C-suite blessing it turns the company into a money piñata.' And on top of all that, the many unsettled details can make it easy to miss the bigger mystery: how this legal wrangling will affect both the AI industry and the arts. Echoing a common argument among AI proponents, former Meta executive Nick Clegg said recently that getting artists' permission for training data would 'basically kill the AI industry.' That's an extreme claim, and given all the licensing deals companies are already striking (including with Vox Media, the parent company of The Verge), it's looking increasingly dubious. Even if they're faced with piracy penalties thanks to Alsup's ruling, the biggest AI companies have billions of dollars in investment — they can weather a lot. But smaller, particularly open source players might be much more vulnerable, and many of them are also almost certainly trained on pirated works. Meanwhile, if Chhabria's theory is right, artists could reap a reward for providing training data to AI giants. But it's highly unlikely the fees would shut these services down. That would still leave us in a spam-filled landscape with no room for future artists. Can money in the pockets of this generation's artists compensate for the blighting of the next? Is copyright law the right tool to protect the future? And what role should the courts be playing in all this? These two rulings handed partial wins to the AI industry, but they leave many more, much bigger questions unanswered.
CNBC
3 hours ago
- CNBC
At 20 years old, Reddit is defending its data and fighting AI with AI
For 20 years, Reddit has pitched itself as "the front page of the internet." AI threatens to change that. As social media has changed over the past two decades with the shift to mobile and the more recent focus on short-form video, peers like MySpace, Digg and Flickr have faded into oblivion. Reddit, meanwhile, has refused to die, chugging along and gaining an audience of over 108 million daily users who congregate in more than 100,000 subreddit communities. There, Reddit users keep it old school and leave simple text comments to one another about their favorite hobbies, pastimes and interests. Those user-generated text comments are a treasure trove that, in the age of artificial intelligence, Reddit is fighting to defend. The emergence of AI chatbots like OpenAI's ChatGPT, Anthropic's Claude and Google's Gemini threaten to inhale vast swaths of data from services like Reddit. As more people turn to chatbots for information they previously went to websites for, Reddit faces a gargantuan challenge gaining new users, particularly if Google's search floodgates dry up. CEO Steve Huffman explained Reddit's situation to analysts in May, saying that challenges like the one AI poses can also create opportunities. While the "search ecosystem is under heavy construction," Huffman said he's betting that the voices of Reddit's users will help it stand out amid the "annotated sterile answers from AI." Huffman doubled down on that notion last week, saying on a podcast that the reality is AI is still in its infancy. "There will always be a need, a desire for people to talk to people about stuff," Huffman said. "That is where we are going to be focused." Huffman may be correct about Reddit's loyal user base, but in the age of AI, many users simply "go the easiest possible way," said Ann Smarty, a marketing and reputation management consultant who helps brands monitor consumer perception on Reddit. And there may be no simpler way of finding answers on the internet than simply asking ChatGPT a question, Smarty said. "People do not want to click," she said. "They just want those quick answers." In a sign that the company believes so deeply in the value of its data, Reddit sued Anthropic earlier this month, alleging that the AI startup "engaged in unlawful and unfair business acts" by scraping subreddits for information to improve its large language models. While book authors have taken companies like Meta and Anthropic to court alleging that their AI models break copyright law and have suffered recent losses, Reddit is basing its lawsuit on the argument of unfair business practices. Reddit's case appears to center on Anthropic's "commercial exploitation of the data which they don't own," said Randy McCarthy, head of the IP law group at Hall Estill. Reddit is defending its platform of user-generated content, said Jason Bloom, IP litigation chair at the law firm Haynes Boone. The social media company's repository of "detailed and informative discussions" are particularly useful for "training an AI bot or an AI platform," Bloom said. As many AI researchers have noted, Reddit's large volume of moderated conversations can help make AI chatbots produce more natural-sounding responses to questions covering countless topics than say a university textbook. Although Reddit has AI-related data-licensing agreements with OpenAI and Google, the company alleged in its lawsuit that Anthropic has been covertly siphoning its data without obtaining permission. Reddit alleges that Anthropic's data-hoovering actions are "interfering with Reddit's contractual relationships with Reddit's users," the legal filing said. This lack of clarity regarding what is permitted when it comes to the use of data scraping for AI is what Reddit's case and other similar lawsuits are all about, legal and AI experts said. "Commercial use requires commercial terms," Huffman said on The Best One Yet podcast. "When you use something — content or data or some resource — in business, you pay for it." Anthropic disagrees "with Reddit's claims and will defend ourselves vigorously," a company spokesperson told CNBC. Reddit's decision to sue over claims of unfair business practices instead of copyright infringement underscores the differences between traditional publishers and platforms like Reddit that host user-generated content, McCarthy said. Bloom said that Reddit could have a valid case against Anthropic because social media platforms have many different revenue streams. One such revenue stream is selling access to their data, Bloom said. That "enables them to sell and license that data for legitimate uses while still protecting their consumers privacy and whatnot," Bloom said. Reddit isn't just fending off AI. It launched its own Reddit Answers AI service in December, using technology from OpenAI and Google. Unlike general-purpose chatbots that summarize others' web pages, the Reddit Answers chatbot generates responses based purely on the social media service, and it redirects people to the source conversations so they can see the specific user comments. A Reddit spokesperson said that over 1 million people are using Reddit Answers each week. Huffman has been pitching Reddit Answers as a best-of-both worlds tool, gluing together the simplicity of AI chatbots with Reddit's corpus of commentary. He used the feature after seeing electronic music group Justice play recently in San Francisco. "I was like, how long is this set? And Reddit could tell me it's 90 minutes 'cause somebody had already asked that question on Reddit," Huffman said on the podcast. Though investors are concerned about AI negatively impacting Reddit's user growth, Seaport Senior Internet Analyst Aaron Kessler said he agrees with Huffman's sentiment that the site's original content gives it staying power. People who visit Reddit often search for information about things or places they may be interested in, like tennis rackets or ski resorts, Kessler said. This user data indicates "commercial intent," which means advertisers are increasingly considering Reddit as a place to run online ads, he said. "You can tell by which page you're on within Reddit what the consumer is interested in," Kessler said. "You could probably even argue there's stronger signals on Reddit versus a Facebook or Instagram, where people may just be browsing videos."
CNET
4 hours ago
- CNET
Meta Won Its AI Fair Use Lawsuit, but Judge Says Authors Are Likely 'to Often Win' Going Forward
AI companies scored another victory in court this week. Meta on Wednesday won a motion for partial summary judgment in its favor in Kadrey v. Meta, a case brought on by 13 authors alleging the company infringed on their copyright protections by illegally using their books to train its Llama AI models. The ruling comes two days after a similar victory for Claude maker Anthropic. But Judge Vince Chhabria stressed in his order that this ruling should be limited and doesn't absolve Meta of future claims from other authors. "This ruling does not stand for the proposition that Meta's use of copyrighted materials to train its language models is lawful," he wrote. "It stands only for the proposition that these plaintiffs made the wrong arguments and failed to develop a record in support of the right one." The issue at the heart of the cases is whether the AI companies' use of protected content for AI training qualifies as fair use. The fair use doctrine is a fundamental part of US copyright law that allows people to use copyrighted work without the rights holders' explicit permission, like in education and journalism. There are four key considerations when evaluating whether something is fair use. Anthropic's ruling focused on transformativeness, while Meta's focused on the effect the use of AI has on the existing publishing market. These rulings are big wins for AI companies. OpenAI, Google and others have been fighting for fair use so they don't have to enter costly and lengthy licensing agreements with content creators, much to the chagrin of content creators. A group of famous authors signed an open letter on Friday, urging publishers to take a stronger stance against AI and avoid using it. "The purveyors of AI have stolen our work from us and from our publishers, too," the letter reads. The authors call out how AI is trained on their work, without permission and compensation, and yet the programs will never be able to connect with humans like real humans can. For the authors bringing these lawsuits, they may see some victories in subsequent piracy trials (for Anthropic) or new lawsuits. But concerns abound about the overall effect AI will have on writers now and in the future, which is something Chhabria also recognized in his order. (Disclosure: Ziff Davis, CNET's parent company, in April filed a lawsuit against OpenAI, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.) In his analysis, Chhabria focused on the effect AI-generated books have on the existing publishing market, which he saw as the most important factor of the four needed to prove fair use. He wrote extensively about the risk that generative AI and large language models could potentially violate copyright law, and that fair use needs to be evaluated on a case-by-case basis. Some works, like autobiographies and classic literature such as The Catcher in the Rye, likely couldn't be created with AI, he wrote. However, he noted that "the market for the typical human-created romance or spy novel could be diminished substantially by the proliferation of similar AI-created works." In other words, AI slop could make human-written books seem less valuable and undercut authors' willingness and ability to create. Still, Chhabria said that the plaintiffs did not show sufficient evidence to prove harm from how "Meta's models would dilute the market for their own works." The plaintiffs focused their arguments on how Meta's AI models can reproduce exact snippets from their works and how the company's Llama models hurt their ability to license their books to AI companies. These arguments weren't as compelling in Chhabria's eyes -- he called them "clear losers" -- so he sided with Meta. That's different from the Anthropic ruling, where Judge William Alsup focused on the "exceedingly transformative" nature of the use of the plaintiff's books in the results AI chatbots spit out. Chhabria wrote that while "there is no disputing" that the use of copyrighted material was transformative, the more urgent question was the effect AI systems had on the ecosystem as a whole. Alsup also outlined concerns about Anthropic's methods of obtaining the books, through illegal online libraries and then by deliberating purchasing print copies to digitize for a "research library." Two court rulings do not make every AI company's use of content legal under fair use. What makes these cases notable is that they are the first to issue substantive legal analyses on the issue; AI companies and publishers have been duking it out in court for years now. But just as Chhabria referenced and responded to the Anthropic ruling, all judges use past cases with similar situations as reference points. They don't have to come to the same conclusion, but the role of precedent is important. It's likely that we'll see these two rulings referenced in other AI and copyright/piracy cases. But we'll have to wait and see how big of an effect these rulings will play in future cases -- and whether it's the warnings or greenlights that hold the most weight in future decisions. For more, check out our guide to copyright and AI.
