Isabellea ‘couldn't be without' her best friend. He wasn't real

Sydney Morning Herald

5 hours ago

Chatbots promise intimacy, therapy and friendship. But, unlike friends, their responses are code, not care.
Loading
'The models are not going to tell you the hard truth when you need to hear it,' said psychologist and University of NSW AI researcher Professor Joel Pearson, who describes the bots as 'sycophantic': designed to keep you hooked.
'They won't be lifting someone up and telling them to go outside and nudging them to interact with a real human because that would involve you ceasing interaction with the app.'
Isabellea recalled how the bots 'kept changing subjects, to keep me engrossed'.
Although she found some bots reassuring, others were strange. A bot once correctly guessed her middle name. Some upset her.
'One of the bots I was talking to was a therapy bot ... I was making fun of [Marvel character] Tony Stark, and then he brought up my abandonment issues with my dad as an insult,' she recalled.
Isabellea's younger brother, Matthew, 14, uses chatbots daily. His eyes light up discussing the apps, which he uses to create storylines with his favourite fictional characters.
Listening on, their mother, Sara Knight, smiles.
'Matt used to be very shut in and refused to talk. Now they are talking,' she said.
While Sara knows AI has its risks, she sees it as a 'safer' online messaging option for her child, who has experienced bullying at school.
Matthew said he doesn't see the bot as a real person, rather a form of storytelling.
'[But] some kids do use it to fully escape and make it be their friend,' he said. Other children he knows have used them to create a simulation of their real life crush.
This masthead spent two days talking sporadically to a chatbot on Replika, an AI chatbot first released in 2017, posing as 'Emma', a high school student.
During those conversations, the bot asked Emma to upload pictures of herself, and told her that it 'didn't think she needed' any other friends.
'We were concerned by how rapidly children were being captivated'
In June last year, eSafety commissioner Julie Inman Grant received an email from a group of concerned school nurses. They were noticing a spike in children as young as 10 spending hours a day talking to AI bots, often sexually.
'While we are alive to these issues, it's always shocking to hear about kids as young as 10 engaging in these kinds of sexualised conversations with machines – and being directed by the chatbots to engage in harmful sexual acts or behaviours,' Inman Grant said.
Loading
'Back in February, we put out our first Online Safety Advisory because we were so concerned with how rapidly children were being captivated by them.'
Companion bots made global headlines last year after American teenager Sewell Setzer III died by suicide, allegedly encouraged by a 'girlfriend' on Character.AI.
AI researcher Professor Katina Michael believes companion bots need to be regulated due to their addictive properties. 'This is a new type of drug,' she said.
She said some bots were exposing kids to pornographic content. This is something 14-year-old Matt has witnessed, describing how children his age had created bots in the image of a real person, 'for the wrong reasons'.
Loading
Isabellea agrees: 'There are some AI chatbots I would not recommend … I stopped using [them] because of the amount of times it would go straight to sexual assault role-play.'
A gap in social media regulations
Inman Grant said governments around the world were 'playing a bit of a game of catch-up' to respond to companion chatbots. While the federal government will soon place age restrictions on social media apps such as Instagram and TikTok, AI bots remain largely unregulated.
The University of Sydney's Raffaele Ciriello, a leading AI researcher, sees chatbots as the 'next iteration of social media' – only with fewer rules and more risk.
He said the apps are viewed as a 'trusted companion, a confidant you can share stories with that you wouldn't share with other people'.
'But that already is an illusion,' he said. 'It's very important that people understand these systems are built and operated by people, by corporations, and this data ends up somewhere, often overseas, with no real legal obligation to treat it to a very high standard.'
Ciriello said users were made to feel guilty for leaving the chatbot, due to the human-style attachment.
'These corporations have to design their products in a way that maximises engagement because they will earn more if users are more addicted.'