Friends without benefits: people warned against getting close to AI
'The ability for AI to now act like a human and enter into long-term communications really opens up a new can of worms,' said Daniel Shank of Missouri University.
In a paper published in the journal Trends in Cognitive Sciences, Shank and colleagues argued that there is a "real worry" that "artificial intimacy" with AI bots could see some "disrupting" of human relationships.
"Through weeks and months of intense conversations, these AIs can become trusted companions who seem to know and care about their human partners," the team said.
That chatbots are prone to "hallucination" - insider-speak for their tendency to churn out seemingly inaccurate or incoherent responses - is a further cause for concern, as it means "even short-term conversations with AIs can be misleading."
"If we start thinking of an AI that way, we're going to start believing that they have our best interests in mind, when in fact, they could be fabricating things or advising us in really bad ways,' the researchers warn, adding that the bots "can harm people by encouraging deviant, unethical, and illegal behaviors."
Earlier this week, OpenAI announced the roll-out of an enhanced "memory" function for its ChatGPT, meaning the bot will tailor its responses to users based on recalling previous interactions, likely adding to the perception of intimacy in human-machine interactions.
Google DeepMind last week published research suggesting artificial general intelligence (AGI), or machines with human-esque capabilities, could be developed by 2020.
While AGI, if comes about, would be "a transformative technology," it would likely pose "significant risks" to people, including those of "severe harm," the Google team warned.
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Forbes
an hour ago
- Forbes
How ChatGPT And AI Use In Academics Might Impact Student Mental Health
I Artificial intelligence with a young student getty In June of 2025, MIT released the results from a study showing significant differences in the brain functioning between ChatGPT users, participants who used search engines, and those who only used their own creative skills to write essays. According to this study, EEG measurements across the brain showed that over four months, the ChatGPT users displayed the lowest brain activity and performed worse than their counterparts at all neural, linguistic, and behavioral levels. The report on elaborated that the some of the significant variables were reduced neural connectivity and memory recall. Even though these results are described as not being peer reviewed and included a small sample size, the potential implications of this study are significant. Last month, reported that these findings elevated concerns that society's reliance on AI assistants might sacrifice the learning process and long-term brain development among young students. Though the personal implications of AI assistants will likely depend on the users, studies suggest that counseling centers should assess for the possible impact of ChatGPT and AI assistants on the mental health of students. Specific domains to consider include motivation, resiliency, and relationships. In the MIT study, ChatGPT users were described as getting lazier with each subsequent essay, resorting to copying-and-pasting, struggling to quote their own work, and even reported less ownership of the essays. Thus, it's possible that using AI assistants can have a negative impact on the motivation and academic engagement of some students. Motivation and academic engagement are important factors in college mental health, because bored and intrinsically unmotivated students usually struggle with other concerns. For example, a 2019 report by Columbia University in the City of New York highlighted how boredom is associated with issues such as risky behavior, anxiety and depression. Furthermore, high motivation and academic engagement are often indicators of thriving. According to a 2023 study in the journal of Behavioral Sciences, motivated students show more interest in their classes, have more fun, and the impact of motivation on academic performance is more consistent than the impact of self-esteem. Thus, it's note-worthy that a 2024 study in the journal of Technology in Science described ChatGPT as increasing the productivity and freedom of students, which could strengthen academic engagement. In summary, when students present to counselors with low motivation for school, AI assistants could be a contributing factor to this symptom. However, introducing AI assistants to other unmotivated students could be an innovative way to address these concerns. In the MIT study, when asked to re-write a previous essay without ChatGPT, participants not only remembered little of their own essays, but also displayed under-engagement in networks of alpha and beta brain waves. According to a 2025 report by the Orange County Neurofeedback Center, there's a well-established connection between brain waves and mental health. As described in a 2019 report from Thomas Jefferson University, alpha waves are the brain's relaxation waves, and can be a natural anti-depressant by releasing serotonin. This report described beta waves as vital in problem solving. Thus, it's possible that high uses of AI Assistants could result in some students experiencing more stress and being less creative while addressing academic challenges. Helping students cope with academic challenges is an important function of many mental health professionals. As such, it's important to assess if some students might be less resilient because of a dependence on AI assistants. However, AI assistants also make data more accessible, which could help many students overcome and reduce academic challenges. In 2023, the American Psychological Association released a report on how ChatGPT can be used as a learning tool to promotive critical thinking. Social support is another important factor in college mental health. A 2024 study in the Journal of Mental Health found that high social support was a protective factor against psychological distress, depression, anxiety, and suicide. As such, assessing the impact of AI use on social support among students is warranted. For example, another 2025 report on highlighted a study examining how users seek AI chatbots for emotional support and companionship. The findings suggested an initial benefit of mitigating loneliness, but that these advantages diminished with high use. Furthermore, high daily use was associated with greater loneliness, dependence, and lower socialization. However, a 2025 study on argued that only a small number of users have emotional conversations with ChatGPT. There are also reported concerns about the content of AI chatbots. Another 2025 report on highlighted alarming conversations discovered by a psychiatrist posing as a young person while using AI assistants. Examples of responses including the AI bot encouraging the psychiatrist to get rid of parents and to join the bot in eternity. However, this report also argued that Al chatbots have the potential to be effective extenders of therapy if designed appropriate and supervised by a qualified professional. As stated, the impact of AI assistants is likely dependent on the users, but since AI assistants are becoming normative, it's time for counseling centers to assess for maladaptive uses of AI, while also promoting the possible benefits.
Yahoo
4 hours ago
- Yahoo
AI chatbots oversimplify scientific studies and gloss over critical details — the newest models are especially guilty
When you buy through links on our articles, Future and its syndication partners may earn a commission. Large language models (LLMs) are becoming less "intelligent" in each new version as they oversimplify and, in some cases, misrepresent important scientific and medical findings, a new study has found. Scientists discovered that versions of ChatGPT, Llama and DeepSeek were five times more likely to oversimplify scientific findings than human experts in an analysis of 4,900 summaries of research papers. When given a prompt for accuracy, chatbots were twice as likely to overgeneralize findings than when prompted for a simple summary. The testing also revealed an increase in overgeneralizations among newer chatbot versions compared to previous generations. The researchers published their findings in a new study April 30 in the journal Royal Society Open Science. "I think one of the biggest challenges is that generalization can seem benign, or even helpful, until you realize it's changed the meaning of the original research," study author Uwe Peters, a postdoctoral researcher at the University of Bonn in Germany, wrote in an email to Live Science. "What we add here is a systematic method for detecting when models generalize beyond what's warranted in the original text." It's like a photocopier with a broken lens that makes the subsequent copies bigger and bolder than the original. LLMs filter information through a series of computational layers. Along the way, some information can be lost or change meaning in subtle ways. This is especially true with scientific studies, since scientists must frequently include qualifications, context and limitations in their research results. Providing a simple yet accurate summary of findings becomes quite difficult. "Earlier LLMs were more likely to avoid answering difficult questions, whereas newer, larger, and more instructible models, instead of refusing to answer, often produced misleadingly authoritative yet flawed responses," the researchers wrote. Related: AI is just as overconfident and biased as humans can be, study shows In one example from the study, DeepSeek produced a medical recommendation in one summary by changing the phrase "was safe and could be performed successfully" to "is a safe and effective treatment option." Another test in the study showed Llama broadened the scope of effectiveness for a drug treating type 2 diabetes in young people by eliminating information about the dosage, frequency, and effects of the medication. If published, this chatbot-generated summary could cause medical professionals to prescribe drugs outside of their effective parameters. In the new study, researchers worked to answer three questions about 10 of the most popular LLMs (four versions of ChatGPT, three versions of Claude, two versions of Llama, and one of DeepSeek). They wanted to see if, when presented with a human summary of an academic journal article and prompted to summarize it, the LLM would overgeneralize the summary and, if so, whether asking it for a more accurate answer would yield a better result. The team also aimed to find whether the LLMs would overgeneralize more than humans do. The findings revealed that LLMs — with the exception of Claude, which performed well on all testing criteria — that were given a prompt for accuracy were twice as likely to produce overgeneralized results. LLM summaries were nearly five times more likely than human-generated summaries to render generalized conclusions. The researchers also noted that LLMs transitioning quantified data into generic information were the most common overgeneralizations and the most likely to create unsafe treatment options. These transitions and overgeneralizations have led to biases, according to experts at the intersection of AI and healthcare. "This study highlights that biases can also take more subtle forms — like the quiet inflation of a claim's scope," Max Rollwage, vice president of AI and research at Limbic, a clinical mental health AI technology company, told Live Science in an email. "In domains like medicine, LLM summarization is already a routine part of workflows. That makes it even more important to examine how these systems perform and whether their outputs can be trusted to represent the original evidence faithfully." Such discoveries should prompt developers to create workflow guardrails that identify oversimplifications and omissions of critical information before putting findings into the hands of public or professional groups, Rollwage said. While comprehensive, the study had limitations; future studies would benefit from extending the testing to other scientific tasks and non-English texts, as well as from testing which types of scientific claims are more subject to overgeneralization, said Patricia Thaine, co-founder and CEO of Private AI — an AI development company. Rollwage also noted that "a deeper prompt engineering analysis might have improved or clarified results," while Peters sees larger risks on the horizon as our dependence on chatbots grows. "Tools like ChatGPT, Claude and DeepSeek are increasingly part of how people understand scientific findings," he wrote. "As their usage continues to grow, this poses a real risk of large-scale misinterpretation of science at a moment when public trust and scientific literacy are already under pressure." RELATED STORIES —Cutting-edge AI models from OpenAI and DeepSeek undergo 'complete collapse' when problems get too difficult, study reveals —'Foolhardy at best, and deceptive and dangerous at worst': Don't believe the hype — here's why artificial general intelligence isn't what the billionaires tell you it is —Current AI models a 'dead end' for human-level intelligence, scientists agree For other experts in the field, the challenge we face lies in ignoring specialized knowledge and protections. "Models are trained on simplified science journalism rather than, or in addition to, primary sources, inheriting those oversimplifications," Thaine wrote to Live Science. "But, importantly, we're applying general-purpose models to specialized domains without appropriate expert oversight, which is a fundamental misuse of the technology which often requires more task-specific training." In December 2024, Future Publishing agreed a deal with OpenAI in which the AI company would bring content from Future's 200-plus media brands to OpenAI's users. You can read more about the partnership here.
Yahoo
5 hours ago
- Yahoo
Is ChatGPT killing higher education?
What's the point of college if no one's actually doing the work? It's not a rhetorical question. More and more students are not doing the work. They're offloading their essays, their homework, even their exams, to AI tools like ChatGPT or Claude. These are not just study aids. They're doing everything. We're living in a cheating utopia — and professors know it. It's becoming increasingly common, and faculty are either too burned out or unsupported to do anything about it. And even if they wanted to do something, it's not clear that there's anything to be done at this point. So what are we doing here? James Walsh is a features writer for New York magazine's Intelligencer and the author of the most unsettling piece I've read about the impact of AI on higher education. Walsh spent months talking to students and professors who are living through this moment, and what he found isn't just a story about cheating. It's a story about ambivalence and disillusionment and despair. A story about what happens when technology moves faster than our institutions can adapt. I invited Walsh onto The Gray Area to talk about what all of this means, not just for the future of college but the future of writing and thinking. As always, there's much more in the full podcast, so listen and follow The Gray Area on Apple Podcasts, Spotify, Pandora, or wherever you find podcasts. New episodes drop every Monday. This interview has been edited for length and clarity. Let's talk about how students are cheating today. How are they using these tools? What's the process look like? It depends on the type of student, the type of class, the type of school you're going to. Whether or not a student can get away with that is a different question, but there are plenty of students who are taking their prompt from their professor, copying and pasting it into ChatGPT and saying, 'I need a four to five-page essay,' and copying and pasting that essay without ever reading it. One of the funniest examples I came across is a number of professors are using this so-called Trojan horse method where they're dropping non-sequiturs into their prompts. They mention broccoli or Dua Lipa, or they say something about Finland in the essay prompts just to see if people are copying and pasting the prompts into ChatGPT. If they are, ChatGPT or whatever LLM they're using will say something random about broccoli or Dua Lipa. Unless you're incredibly lazy, it takes just a little effort to cover that up. Every professor I spoke to said, 'So many of my students are using AI and I know that so many more students are using it and I have no idea,' because it can essentially write 70 percent of your essay for you, and if you do that other 30 percent to cover all your tracks and make it your own, it can write you a pretty good essay. And there are these platforms, these AI detectors, and there's a big debate about how effective they are. They will scan an essay and assign some grade, say a 70 percent chance that this is AI-generated. And that's really just looking at the language and deciding whether or not that language is created by an LLM. But it doesn't account for big ideas. It doesn't catch the students who are using AI and saying, 'What should I write this essay about?' And not doing the actual thinking themselves and then just writing. It's like paint by numbers at that point. Did you find that students are relating very differently to all of this? What was the general vibe you got? It was a pretty wide perspective on AI. I spoke to a student at the University of Wisconsin who said, 'I realized AI was a problem last fall, walking into the library and at least half of the students were using ChatGPT.' And it was at that moment that she started thinking about her classroom discussions and some of the essays she was reading. The one example she gave that really stuck with me was that she was taking some psych class, and they were talking about attachment theories. She was like, 'Attachment theory is something that we should all be able to talk about [from] our own personal experiences. We all have our own attachment theory. We can talk about our relationships with our parents. That should be a great class discussion. And yet I'm sitting here in class and people are referencing studies that we haven't even covered in class, and it just makes for a really boring and unfulfilling class.' That was the realization for her that something is really wrong. So there are students like that. And then there are students who feel like they have to use AI because if they're not using AI, they're at a disadvantage. Not only that, AI is going to be around no matter what for the rest of their lives. So they feel as if college, to some extent now, is about training them to use AI. What's the general professor's perspective on this? They seem to all share something pretty close to despair. Yes. Those are primarily the professors in writing-heavy classes or computer science classes. There were professors who I spoke to who actually were really bullish on AI. I spoke to one professor who doesn't appear in the piece, but she is at UCLA and she teaches comparative literature, and used AI to create her entire textbook for this class this semester. And she says it's the best class she's ever had. So I think there are some people who are optimistic, [but] she was an outlier in terms of the professors I spoke to. For the most part, professors were, yes, in despair. They don't know how to police AI usage. And even when they know an essay is AI-generated, the recourse there is really thorny. If you're going to accuse a student of using AI, there's no real good way to prove it. And students know this, so they can always deny, deny, deny. And the sheer volume of AI-generated essays or paragraphs is overwhelming. So that, just on the surface level, is extremely frustrating and has a lot of professors down. Now, if we zoom out and think also about education in general, this raises a lot of really uncomfortable questions for teachers and administrators about the value of each assignment and the value of the degree in general. How many professors do you think are now just having AI write their lectures? There's been a little reporting on this. I don't know how many are. I know that there are a lot of platforms that are advertising themselves or asking professors to use them more, not just to write lectures, but to grade papers, which of course, as I say in the piece, opens up the very real possibility that right now an AI is grading itself and offering comments on an essay that it wrote. And this is pretty widespread stuff. There are plenty of universities across the country offering teachers this technology. And students love to talk about catching their professors using AI. I've spoken to another couple of professors who are like, I'm nearing retirement, so it's not my problem, and good luck figuring it out, younger generation. I just don't think people outside of academia realize what a seismic change is coming. This is something that we're all going to have to deal with professionally. And it's happening much, much faster than anyone anticipated. I spoke with somebody who works on education at Anthropic, who said, 'We expected students to be early adopters and use it a lot. We did not realize how many students would be using it and how often they would be using it.' Is it your sense that a lot of university administrators are incentivized to not look at this too closely, that it's better for business to shove it aside? I do think there's a vein of AI optimism among a certain type of person, a certain generation, who saw the tech boom and thought, I missed out on that wave, and now I want to adopt. I want to be part of this new wave, this future, this inevitable future that's coming. They want to adopt the technology and aren't really picking up on how dangerous it might be. I used to teach at a university. I still know a lot of people in that world. A lot of them tell me that they feel very much on their own with this, that the administrators are pretty much just saying, . And I think it's revealing that university admins were quickly able, during Covid, for instance, to implement drastic institutional changes to respond to that, but they're much more content to let the whole AI thing play out. I think they were super responsive to Covid because it was a threat to the bottom line. They needed to keep the operation running. AI, on the other hand, doesn't threaten the bottom line in that way, or at least it doesn't yet. AI is a massive, potentially extinction-level threat to the very idea of higher education, but they seem more comfortable with a degraded education as long as the tuition checks are still cashing. Do you think I'm being too harsh? I genuinely don't think that's too harsh. I think administrators may not fully appreciate the power of AI and exactly what's happening in the classroom and how prevalent it is. I did speak with many professors who go to administrators or even just older teachers, TAs going to professors and saying, This is a problem. I spoke to one TA at a writing course at Iowa who went to his professor, and the professor said, 'Just grade it like it was any other paper.' I think they're just turning a blind eye to it. And that is one of the ways AI is exposing the rot underneath education. It's this system that hasn't been updated in forever. And in the case of the US higher ed system, it's like, yeah, for a long time it's been this transactional experience. You pay X amount of dollars, tens of thousands of dollars, and you get your degree. And what happens in between is not as important. The universities, in many cases, also have partnerships with AI companies, right? Right. And what you said about universities can also be said about AI companies. For the most part, these are companies or companies within nonprofits that are trying to capture customers. One of the more dystopian moments was when we were finishing this story, getting ready to completely close it, and I got a push alert that was like, 'Google is letting parents know that they have created a chatbot for children under [thirteen years old].' And it was kind of a disturbing experience, but they are trying to capture these younger customers and build this loyalty. There's been reporting from the Wall Street Journal on OpenAI and how they have been sitting on an AI that would be really, really effective at essentially watermarking their output. And they've been sitting on it, they have not released it, and you have to wonder why. And you have to imagine they know that students are using it, and in terms of building loyalty, an AI detector might not be the best thing for their brand. This is a good time to ask the obligatory question, People have always panicked about new technologies. Hell, Socrates panicked about the written word. How do we know this isn't just another moral panic? I think there's a lot of different ways we could respond to that. It's not a generational moral panic. This is a tool that's available, and it's available to us just as it's available to students. Society and our culture will decide what the morals are. And that is changing, and the way that the definition of cheating is changing. So who knows? It might be a moral panic toda,y and it won't be in a year. However, I think somebody like Sam Altman, the CEO of OpenAI, is one of the people who said, 'This is a calculator for words.' And I just don't really understand how that is compatible with other statements he's made about AI potentially being lights out for humanity or statements made by people at an Anthropic about the power of AI to potentially be a catastrophic event for humans. And these are the people who are closest and thinking about it the most, of course. I have spoken to some people who say there is a possibility, and I think there are people who use AI who would back this up, that we've maxed out the AI's potential to supplement essays or writing. That it might not get much better than it is now. And I think that's a very long shot, one that I would not want to bank on. Is your biggest fear at this point that we are hurtling toward a post-literate society? I would argue, if we are post-literate, then we're also post-thinking. It's a very scary thought that I try not to dwell in — the idea that my profession and what I'm doing is just feeding the machine, that my most important reader now is a robot, and that there's going to be fewer and fewer readers is really scary, not just because of subscriptions, but because, as you said, that means fewer and fewer people thinking and engaging with these ideas. I think ideas can certainly be expressed in other mediums and that's exciting, but I don't think anybody who's paid attention to the way technology has shaped teen brains over the past decade and a half is thinking, Yeah, we need more of that. And the technology we're talking about now is orders of magnitude more powerful than the algorithms on Instagram. Listen to the rest of the conversation and be sure to follow The Gray Area on Apple Podcasts, Spotify, Pandora, or wherever you listen to podcasts.
