Beware! Research shows Gmail's AI email summaries can be hacked

Android Authority

2 days ago

Edgar Cervantes / Android Authority
TL;DR A researcher recently demonstrated a Gemini flaw that could be exploited to inject malicious instructions while using Gmail's email summary feature.
These instructions were hidden in plain text under the body of the email.
Google responded to the research, stating that it had updated its models to identify such prompt engineering measures and block phishing links.
Big tech companies have been billing AI as the ubiquitous tool that frees us from mundane activities, and that includes reading long emails thoroughly. But little do we hear about the possibility of AI unknowingly leading us into traps that may be used to steal our sensitive data. That's precisely what recent research highlighted when it discussed the possibility of hackers using Gemini as means for phishing.
Recently, a cybersecurity researcher demonstrated a vulnerability targeting Google Workspace users where Gemini can be manipulated to display malicious instructions. The vulnerability was submitted to 0din, which is the Mozilla Foundation's bug bounty program for AI applications, and talks more specifically about the ease of misguiding Gmail's email summarization feature for Google Workspace subscribers.
The submission demonstrates how deceptive prompts can be inserted into an email's body in plain HTML format or as text hidden with an invisible font color. Gemini interprets these prompts as commands and can display them in the email summary without any caution.
Since the message is hidden in the body of the original email, it goes unnoticed by the receiver, who is likely to believe it to be a warning generated by Gemini. Researcher blurrylogic pointed out that this can be exploited to display messages that may compel the recipient to share sensitive information without proper verification, which could lead to their credentials being stolen using social engineering.
Shortly after the findings were published on 0din, Google shared details about steps it had taken to make Gemini more resilient against such tactics. Addressing reports about Gemini's vulnerability, Google said it continually updates its repository of malicious prompts or instructions that can manipulate the chatbot's output. The underlying machine learning models are constantly trained to ensure they don't respond to malicious instructions.
Google
Google also listed other steps it takes to counter different forms of phishing attempts. It noted that Gemini identifies suspicious or rogue links disguised as useful ones in the email body and redacts them from the email summaries. To further strengthen its security measures, Gemini also requests confirmation for actions such as deleting specific tasks.
Despite Google's prompt measures, we should be warned that online threat perpetrators usually think one step ahead. Therefore, we advise against blindly trusting any messages in Gemini that prompt actions such as clicking a link, making a call, or emailing a specific person.
Got a tip? Talk to us! Email our staff at
Email our staff at [e-mail] . You can stay anonymous or get credit for the info, it's your choice.