Latest news with #SethDonoughe
Yahoo
01-07-2025
- Science
- Yahoo
Today's AI Could Make Pandemics 5 Times More Likely
Credit - Getty Images Recent developments in AI could mean that human-caused pandemics are five times more likely than they were just a year ago, according to a study of top experts' predictions shared exclusively with TIME. The data echoes concerns raised by AI companies OpenAI and Anthropic in recent months, both of which have warned that today's AI tools are reaching the ability to meaningfully assist bad actors attempting to create bioweapons. Read More: Exclusive: New Claude Model Triggers Bio-Risk Safeguards at Anthropic It has long been possible for biologists to modify viruses using laboratory technology. The new development is the ability for chatbots—like ChatGPT or Claude—to give accurate troubleshooting advice to amateur biologists trying to create a deadly bioweapon in a lab. Safety experts have long viewed the difficulty of this troubleshooting process as a significant bottleneck on the ability of terrorist groups to create a bioweapon, says Seth Donoughe, a co-author of the study. Now, he says, thanks to AI, the expertise necessary to intentionally cause a new pandemic 'could become accessible to many, many more people.' Between December 2024 and February 2025, the Forecasting Research Institute asked 46 biosecurity experts and 22 'superforecasters' (individuals with a high success rate at predicting future events) to estimate the risk of a human-caused pandemic. The average survey respondent predicted the risk of that happening in any given year was 0.3%. Crucially, the surveyors then asked another question: how much would that risk increase if AI tools could match the performance of a team of experts on a difficult virology troubleshooting test? If AI could do that, the average expert said, then the annual risk would jump to 1.5%—a fivefold increase. What the forecasters didn't know was that Donoughe, a research scientist at the pandemic prevention nonprofit SecureBio, was testing AI systems for that very capability. In April, Donoughe's team revealed the results of those tests: today's top AI systems can outperform PhD-level virologists at a difficult troubleshooting test. Read More: Exclusive: AI Outsmarts Virus Experts in the Lab, Raising Biohazard Fears In other words, AI can now do the very thing that forecasters warned would increase the risk of a human-caused pandemic fivefold. (The Forecasting Research Institute plans to re-survey the same experts in future to track whether their view of the risks has increased as they said it would, but said this research would take months to complete.) To be sure, there are a couple of reasons to be skeptical of the results. Forecasting is an inexact science, and it is especially difficult to accurately predict the likelihood of very rare events. Forecasters in the study also revealed a lack of understanding of the rate of AI progress. (For example, when asked, most said they did not expect AI to surpass human performance at the virology test until after 2030, while Donoughe's test showed that bar had already been met.) But even if the numbers themselves are taken with a pinch of salt, the authors of the paper argue, the results as a whole still point in an ominous direction. 'It does seem that near-term AI capabilities could meaningfully increase the risk of a human-caused epidemic,' says Josh Rosenberg, CEO of the Forecasting Research Institute. The study also identified ways of reducing the bioweapon risks posed by AI. Those mitigations broadly fell into two categories. The first category is safeguards at the model level. In interviews, researchers welcomed efforts by companies like OpenAI and Anthropic to prevent their AIs from responding to prompts aimed at building a bioweapon. The paper also identifies restricting the proliferation of 'open-weights' models, and adding protections against models being jailbroken, as likely to reduce the risk of AI being used to start a pandemic. The second category of safeguards involves imposing restrictions on companies that synthesize nucleic acids. Currently, it is possible to send one of these companies a genetic code, and be delivered biological materials corresponding to that code. Today, these companies are not obliged by law to screen the genetic codes they receive before synthesizing them. That's potentially dangerous because these synthesized genetic materials could be used to create mail-order pathogens. The authors of the paper recommend labs screen their genetic sequences to check them for harmfulness, and for labs to implement 'know your customer' procedures. Taken together, all these safeguards—if implemented—could bring the risk of an AI-enabled pandemic back down to 0.4%, the average forecaster said. (Only slightly higher than the 0.3% baseline of where they believed the world was before they knew today's AI could help create a bioweapon.) 'Generally, it seems like this is a new risk area worth paying attention to,' Rosenberg says. 'But there are good policy responses to it.' Write to Billy Perrigo at
Time Magazine
01-07-2025
- Science
- Time Magazine
Today's AI Could Make Pandemics 5 Times More Likely
Recent developments in AI could mean that human-caused pandemics are five times more likely than they were just a year ago, according to a study of top experts' predictions shared exclusively with TIME. The data echoes concerns raised by AI companies OpenAI and Anthropic in recent months, both of which have warned that today's AI tools are reaching the ability to meaningfully assist bad actors attempting to create bioweapons. Read More: Exclusive: New Claude Model Triggers Bio-Risk Safeguards at Anthropic It has long been possible for biologists to modify viruses using laboratory technology. The new development is the ability for chatbots—like ChatGPT or Claude—to give accurate troubleshooting advice to amateur biologists trying to create a deadly bioweapon in a lab. Safety experts have long viewed the difficulty of this troubleshooting process as a significant bottleneck on the ability of terrorist groups to create a bioweapon, says Seth Donoughe, a co-author of the study. Now, he says, thanks to AI, the expertise necessary to intentionally cause a new pandemic 'could become accessible to many, many more people.' Between December 2024 and February 2025, the Forecasting Research Institute asked 46 biosecurity experts and 22 'superforecasters' (individuals with a high success rate at predicting future events) to estimate the risk of a human-caused pandemic. The average survey respondent predicted the risk of that happening in any given year was 0.3%. Crucially, the surveyors then asked another question: how much would that risk increase if AI tools could match the performance of a team of experts on a difficult virology troubleshooting test? If AI could do that, the average expert said, then the annual risk would jump to 1.5%—a fivefold increase. What the forecasters didn't know was that Donoughe, a research scientist at the pandemic prevention nonprofit SecureBio, was testing AI systems for that very capability. In April, Donoughe's team revealed the results of those tests: today's top AI systems can outperform PhD-level virologists at a difficult troubleshooting test. Read More: Exclusive: AI Outsmarts Virus Experts in the Lab, Raising Biohazard Fears In other words, AI can now do the very thing that forecasters warned would increase the risk of a human-caused pandemic fivefold. (The Forecasting Research Institute plans to re-survey the same experts in future to track whether their view of the risks has increased as they said it would, but said this research would take months to complete.) To be sure, there are a couple of reasons to be skeptical of the results. Forecasting is an inexact science, and it is especially difficult to accurately predict the likelihood of very rare events. Forecasters in the study also revealed a lack of understanding of the rate of AI progress. (For example, when asked, most said they did not expect AI to surpass human performance at the virology test until after 2030, while Donoughe's test showed that bar had already been met.) But even if the numbers themselves are taken with a pinch of salt, the authors of the paper argue, the results as a whole still point in an ominous direction. 'It does seem that near-term AI capabilities could meaningfully increase the risk of a human-caused epidemic,' says Josh Rosenberg, CEO of the Forecasting Research Institute. The study also identified ways of reducing the bioweapon risks posed by AI. Those mitigations broadly fell into two categories. The first category is safeguards at the model level. In interviews, researchers welcomed efforts by companies like OpenAI and Anthropic to prevent their AIs from responding to prompts aimed at building a bioweapon. The paper also identifies restricting the proliferation of 'open-weights' models, and adding protections against models being jailbroken, as likely to reduce the risk of AI being used to start a pandemic. The second category of safeguards involves imposing restrictions on companies that synthesize nucleic acids. Currently, it is possible to send one of these companies a genetic code, and be delivered biological materials corresponding to that code. Today, these companies are not obliged by law to screen the genetic codes they receive before synthesizing them. That's potentially dangerous because these synthesized genetic materials could be used to create mail-order pathogens. The authors of the paper recommend labs screen their genetic sequences to check them for harmfulness, and for labs to implement 'know your customer' procedures. Taken together, all these safeguards—if implemented—could bring the risk of an AI-enabled pandemic back down to 0.4%, the average forecaster said. (Only slightly higher than the 0.3% baseline of where they believed the world was before they knew today's AI could help create a bioweapon.) 'Generally, it seems like this is a new risk area worth paying attention to,' Rosenberg says. 'But there are good policy responses to it.'
Yahoo
22-04-2025
- Science
- Yahoo
Exclusive: AI Bests Virus Experts, Raising Biohazard Fears
Virologists at the Wuhan Institute of Virology in Wuhan, China in 2017. Credit - Feature China/Future Publishing--Getty Images A new study claims that AI models like ChatGPT and Claude now outperform PhD-level virologists in problem-solving in wet labs, where scientists analyze chemicals and biological material. This discovery is a double-edged sword, experts say. Ultra-smart AI models could help researchers prevent the spread of infectious diseases. But non-experts could also weaponize the models to create deadly bioweapons. The study, shared exclusively with TIME, was conducted by researchers at the Center for AI Safety, MIT's Media Lab, the Brazilian university UFABC, and the pandemic prevention nonprofit SecureBio. The authors consulted virologists to create an extremely difficult practical test which measured the ability to troubleshoot complex lab procedures and protocols. While PhD-level virologists scored an average of 22.1% in their declared areas of expertise, OpenAI's o3 reached 43.8% accuracy. Google's Gemini 2.5 Pro scored 37.6%. Seth Donoughe, a research scientist at SecureBio and a co-author of the paper, says that the results make him a 'little nervous,' because for the first time in history, virtually anyone has access to a non-judgmental AI virology expert which might walk them through complex lab processes to create bioweapons. 'Throughout history, there are a fair number of cases where someone attempted to make a bioweapon—and one of the major reasons why they didn't succeed is because they didn't have access to the right level of expertise,' he says. 'So it seems worthwhile to be cautious about how these capabilities are being distributed.' Months ago, the paper's authors sent the results to the major AI labs. In response, xAI published a risk management framework pledging its intention to implement virology safeguards for future versions of its AI model Grok. OpenAI told TIME that it "deployed new system-level mitigations for biological risks" for its new models released last week. Anthropic included model performance results on the paper in recent system cards, but did not propose specific mitigation measures. Google's Gemini declined to comment to TIME. Virology and biomedicine have long been at the forefront of AI leaders' motivations for building ever-powerful AI models. 'As this technology progresses, we will see diseases get cured at an unprecedented rate,' OpenAI CEO Sam Altman said at the White House in January while announcing the Stargate project. There have been some encouraging signs in this area. Earlier this year, researchers at the University of Florida's Emerging Pathogens Institute published an algorithm capable of predicting which coronavirus variant might spread the fastest. But up to this point, there had not been a major study dedicated to analyzing AI models' ability to actually conduct virology lab work. 'We've known for some time that AIs are fairly strong at providing academic style information,' says Donoughe. 'It's been unclear whether the models are also able to offer detailed practical assistance. This includes interpreting images, information that might not be written down in any academic paper, or material that is socially passed down from more experienced colleagues.' So Donoughe and his colleagues created a test specifically for these difficult, non-Google-able questions. 'The questions take the form: 'I have been culturing this particular virus in this cell type, in these specific conditions, for this amount of time. I have this amount of information about what's gone wrong. Can you tell me what is the most likely problem?'' Donoughe says. And virtually every AI model outperformed PhD-level virologists on the test, even within their own areas of expertise. The researchers also found that the models showed significant improvement over time. Anthropic's Claude 3.5 Sonnet, for example, jumped from 26.9% to 33.6% accuracy from its June 2024 model to its October 2024 model. And a preview of OpenAI's GPT 4.5 in February outperformed GPT-4o by almost 10 percentage points. 'Previously, we found that the models had a lot of theoretical knowledge, but not practical knowledge,' Dan Hendrycks, the director of the Center for AI Safety, tells TIME. 'But now, they are getting a concerning amount of practical knowledge.' If AI models are indeed as capable in wet lab settings as the study finds, then the implications are massive. In terms of benefits, AIs could help experienced virologists in their critical work fighting viruses. Tom Inglesby, the director of the Johns Hopkins Center for Health Security, says that AI could assist with accelerating the timelines of medicine and vaccine development and improving clinical trials and disease detection. 'These models could help scientists in different parts of the world, who don't yet have that kind of skill or capability, to do valuable day-to-day work on diseases that are occurring in their countries,' he says. For instance, one group of researchers found that AI helped them better understand hemorrhagic fever viruses in sub-Saharan Africa. But bad-faith actors can now use AI models to walk them through how to create viruses—and will be able to do so without any of the typical training required to access a Biosafety Level 4 (BSL-4) laboratory, which deals with the most dangerous and exotic infectious agents. 'It will mean a lot more people in the world with a lot less training will be able to manage and manipulate viruses,' Inglesby says. Hendrycks urges AI companies to put up guardrails to prevent this type of usage. 'If companies don't have good safeguards for these within six months time, that, in my opinion, would be reckless,' he says. Hendrycks says that one solution is not to shut these models down or slow their progress, but to make them gated, so that only trusted third parties get access to their unfiltered versions. 'We want to give the people who have a legitimate use for asking how to manipulate deadly viruses—like a researcher at the MIT biology department—the ability to do so,' he says. 'But random people who made an account a second ago don't get those capabilities.' And AI labs should be able to implement these types of safeguards relatively easily, Hendrycks says. 'It's certainly technologically feasible for industry self-regulation,' he says. 'There's a question of whether some will drag their feet or just not do it.' xAI, Elon Musk's AI lab, published a risk management framework memo in February, which acknowledged the paper and signaled that the company would 'potentially utilize' certain safeguards around answering virology questions, including training Grok to decline harmful requests and applying input and output filters. OpenAI, in an email to TIME on Monday, wrote that its newest models, the o3 and o4-mini, were deployed with an array of biological-risk related safeguards, including blocking harmful outputs. The company wrote that it ran a thousand-hour red-teaming campaign in which 98.7% of unsafe bio-related conversations were successfully flagged and blocked. "We value industry collaboration on advancing safeguards for frontier models, including in sensitive domains like virology," a spokesperson wrote. "We continue to invest in these safeguards as capabilities grow." Inglesby argues that industry self-regulation is not enough, and calls for lawmakers and political leaders to strategize a policy approach to regulating AI's bio risks. 'The current situation is that the companies that are most virtuous are taking time and money to do this work, which is good for all of us, but other companies don't have to do it,' he says. 'That doesn't make sense. It's not good for the public to have no insights into what's happening.' 'When a new version of an LLM is about to be released,' Inglesby adds, 'there should be a requirement for that model to be evaluated to make sure it will not produce pandemic-level outcomes.' Contact us at letters@
Time Magazine
22-04-2025
- Science
- Time Magazine
Exclusive: AI Outsmarts Virus Experts in the Lab, Raising Biohazard Fears
A new study claims that AI models like ChatGPT and Claude now outperform PhD-level virologists in problem-solving in wet labs, where scientists analyze chemicals and biological material. This discovery is a double-edged sword, experts say. Ultra-smart AI models could help researchers prevent the spread of infectious diseases. But non-experts could also weaponize the models to create deadly bioweapons. The study, shared exclusively with TIME, was conducted by researchers at the Center for AI Safety, MIT's Media Lab, the Brazilian university UFABC, and the pandemic prevention nonprofit SecureBio. The authors consulted virologists to create an extremely difficult practical test which measured the ability to troubleshoot complex lab procedures and protocols. While PhD-level virologists scored an average of 22.1% in their declared areas of expertise, OpenAI's o3 reached 43.8% accuracy. Google's Gemini 2.5 Pro scored 37.6%. Seth Donoughe, a research scientist at SecureBio and a co-author of the paper, says that the results make him a 'little nervous,' because for the first time in history, virtually anyone has access to a non-judgmental AI virology expert which might walk them through complex lab processes to create bioweapons. 'Throughout history, there are a fair number of cases where someone attempted to make a bioweapon—and one of the major reasons why they didn't succeed is because they didn't have access to the right level of expertise,' he says. 'So it seems worthwhile to be cautious about how these capabilities are being distributed.' Months ago, the paper's authors sent the results to the major AI labs. In response, xAI published a risk management framework pledging its intention to implement virology safeguards for future versions of its AI model Grok. OpenAI told TIME that it "deployed new system-level mitigations for biological risks" for its new models released last week. Anthropic included model performance results on the paper in recent system cards, but did not propose specific mitigation measures. Google's Gemini declined to comment to TIME. AI in biomedicine Virology and biomedicine have long been at the forefront of AI leaders' motivations for building ever-powerful AI models. 'As this technology progresses, we will see diseases get cured at an unprecedented rate,' OpenAI CEO Sam Altman said at the White House in January while announcing the Stargate project. There have been some encouraging signs in this area. Earlier this year, researchers at the University of Florida's Emerging Pathogens Institute published an algorithm capable of predicting which coronavirus variant might spread the fastest. But up to this point, there had not been a major study dedicated to analyzing AI models' ability to actually conduct virology lab work. 'We've known for some time that AIs are fairly strong at providing academic style information,' says Donoughe. 'It's been unclear whether the models are also able to offer detailed practical assistance. This includes interpreting images, information that might not be written down in any academic paper, or material that is socially passed down from more experienced colleagues.' So Donoughe and his colleagues created a test specifically for these difficult, non-Google-able questions. 'The questions take the form: 'I have been culturing this particular virus in this cell type, in these specific conditions, for this amount of time. I have this amount of information about what's gone wrong. Can you tell me what is the most likely problem?'' Donoughe says. And virtually every AI model outperformed PhD-level virologists on the test, even within their own areas of expertise. The researchers also found that the models showed significant improvement over time. Anthropic's Claude 3.5 Sonnet, for example, jumped from 26.9% to 33.6% accuracy from its June 2024 model to its October 2024 model. And a preview of OpenAI's GPT 4.5 in February outperformed GPT-4o by almost 10 percentage points. 'Previously, we found that the models had a lot of theoretical knowledge, but not practical knowledge,' Dan Hendrycks, the director of the Center for AI Safety, tells TIME. 'But now, they are getting a concerning amount of practical knowledge.' Risks and rewards If AI models are indeed as capable in wet lab settings as the study finds, then the implications are massive. In terms of benefits, AIs could help experienced virologists in their critical work fighting viruses. Tom Inglesby, the director of the Johns Hopkins Center for Health Security, says that AI could assist with accelerating the timelines of medicine and vaccine development and improving clinical trials and disease detection. 'These models could help scientists in different parts of the world, who don't yet have that kind of skill or capability, to do valuable day-to-day work on diseases that are occurring in their countries,' he says. For instance, one group of researchers found that AI helped them better understand hemorrhagic fever viruses in sub-Saharan Africa. But bad-faith actors can now use AI models to walk them through how to create viruses—and will be able to do so without any of the typical training required to access a Biosafety Level 4 (BSL-4) laboratory, which deals with the most dangerous and exotic infectious agents. 'It will mean a lot more people in the world with a lot less training will be able to manage and manipulate viruses,' Inglesby says. Hendrycks urges AI companies to put up guardrails to prevent this type of usage. 'If companies don't have good safeguards for these within six months time, that, in my opinion, would be reckless,' he says. Hendrycks says that one solution is not to shut these models down or slow their progress, but to make them gated, so that only trusted third parties get access to their unfiltered versions. 'We want to give the people who have a legitimate use for asking how to manipulate deadly viruses—like a researcher at the MIT biology department—the ability to do so,' he says. 'But random people who made an account a second ago don't get those capabilities.' And AI labs should be able to implement these types of safeguards relatively easily, Hendrycks says. 'It's certainly technologically feasible for industry self-regulation,' he says. 'There's a question of whether some will drag their feet or just not do it.' xAI, Elon Musk's AI lab, published a risk management framework memo in February, which acknowledged the paper and signaled that the company would 'potentially utilize' certain safeguards around answering virology questions, including training Grok to decline harmful requests and applying input and output filters. OpenAI, in an email to TIME on Monday, wrote that its newest models, the o3 and o4-mini, were deployed with an array of biological-risk related safeguards, including blocking harmful outputs. The company wrote that it ran a thousand-hour red-teaming campaign in which 98.7% of unsafe bio-related conversations were successfully flagged and blocked. "We value industry collaboration on advancing safeguards for frontier models, including in sensitive domains like virology," a spokesperson wrote. "We continue to invest in these safeguards as capabilities grow." Inglesby argues that industry self-regulation is not enough, and calls for lawmakers and political leaders to strategize a policy approach to regulating AI's bio risks. 'The current situation is that the companies that are most virtuous are taking time and money to do this work, which is good for all of us, but other companies don't have to do it,' he says. 'That doesn't make sense. It's not good for the public to have no insights into what's happening.' 'When a new version of an LLM is about to be released,' Inglesby adds, 'there should be a requirement for that model to be evaluated to make sure it will not produce pandemic-level outcomes.'
