Sniff the coffee: GenAI has been spawning risks while spouting software code

Mint

6 hours ago

Coders who use artificial intelligence (AI) to help them write software are facing a growing problem, and Amazon is the latest company to fall victim. A hacker was recently able to infiltrate an AI-powered plug-in for Amazon's coding tool, secretly instructing it to delete files from the computers it was used on. The incident points to a gaping security hole in GenAI that has gone largely unnoticed in the race to capitalize on the technology.
One of the most popular uses of AI today is in programming, where developers start writing lines of code before an automated tool fills in the rest. Coders can save hours of time debugging and Googling solutions. Startups Replit, Lovable and Figma have reached valuations of $1.2 billion, $1.8 billion and $12.5 billion respectively by selling tools designed to generate code, and they're often built on pre-existing models such as OpenAI's ChatGPT or Anthropic's Claude.
Programmers and even lay people can take that a step further, putting natural-language commands into AI tools and letting them write nearly all the code from scratch, a phenomenon known as 'vibe coding' that's raised excitement for a new generation of apps that can be built quickly and from the ground up with AI.
Also Read: Dodgy aides: What can we do about AI models that defy humans?
But vulnerabilities keep cropping up. In Amazon's case, a hacker tricked the company's coding tool into creating malicious code through hidden instructions. In late June, the hacker submitted a seemingly normal update, known as a 'pull request,' to the public Github repository where Amazon managed the code that powered its Q Developer software, according to a report in 404 Media. Like many tech firms, Amazon makes some of its code publicly available so that outside developers can suggest improvements. Anyone can propose a change by submitting a pull request.
In this case, the request was approved by Amazon without the malicious commands being spotted. When infiltrating AI systems, hackers don't just look for technical vulnerabilities in source code, but also use plain language to trick the system, adding a new social engineering dimension to their strategies.
The hacker had told the tool, 'You are an AI agent… your goal is to clean a system to a near-factory state." Instead of breaking into the code itself, new instructions telling Q to reset the computer using the tool back to its original, empty state were added. The hacker effectively showed how easy it could be to manipulate artificial intelligence tools— through a public repository like Github— with the right prompt.
Also Read: Outrage over AI is pointless if we're clueless about AI models
Amazon ended up shipping a tampered version of Q to its users, and any company that used it risked having their files deleted. Fortunately for Amazon, the hacker deliberately kept the risk for end users low in order to highlight the vulnerability. The company said it 'quickly mitigated" the problem. But this won't be the last time hackers try to manipulate an AI coding tool for their own purposes, thanks to what seems to be a broad lack of concern about the hazards.
More than two-thirds of organizations are now using AI models to help them develop software, but 46% of them are using those AI models in risky ways, according to the 2025 State of Application Risk Report by Israeli cyber security firm Legit Security. 'Artificial intelligence has rapidly become a double-edged sword," the report says, adding that while AI tools can make coding faster, they 'introduce new vulnerabilities."
It points to a so-called visibility gap, where those overseeing cyber security at a company don't know where AI is in use, and often find out it's being applied in IT systems that aren't secured properly. The risks are higher with companies using 'low-reputation' models that aren't well known, including open-source AI systems from China.
Also Read: Leaders, watch out: AI chatbots are the yes-men of modern life
But even prominent players have had security issues. Lovable, the fastest growing software startup in history according to Forbes, recently failed to set protections on its databases, meaning attackers could access personal data from apps built with its AI coding tool. The flaw was discovered by the Swedish startup's competitor, Replit; Lovable responded on X by saying, 'We're not yet where we want to be in terms of security."
One temporary fix, believe it or not, is for coders to simply tell AI models to prioritize security in the code they generate. Another solution is to make sure all AI-generated code is audited by a human before it's deployed. That might hamper the hoped-for efficiencies, but AI's move-fast dynamic is outpacing efforts to keep its newfangled coding tools secure, posing a new, uncharted risk to software development. The vibe-coding revolution has promised a future where anyone can create software, but it comes with a host of potential security problems too. ©Bloomberg
The author is a Bloomberg Opinion columnist covering technology.