Artificial intelligence a threat to children's safety, say activists

The Star

23-04-2025

Digitally-altered images/videos or deepfakes are threatening the safety of children in Malaysia. Therefore, firm and prompt action must be taken to protect them, a coalition of activists has urged.
'The recent deepfake scandal involving 38 students at a private secondary school in Kulai, Johor, is a stark reminder that threats to our children now extend beyond physical spaces,' said the group, which includes KRYSS Network, All Women's Action Society (Awam) and Women's Aid Organisation (WAO), in a recent statement.
The psychological impact of child sexual abuse – whether online or offline – is devastating and far-reaching, often negatively affecting the future of survivors.
Victims have reported experiencing nightmares, anxiety, depression, suicidal thoughts and deep discomfort attending school alongside individuals complicit in their victimisation, namely those who have actively created, commented or shared the deepfakes.
The youngest survivor is reportedly only 12 to 13 years old. One survivor even shared that a disciplinary teacher victim-blamed her, trivialising the incident and suggesting that the 16-year-old perpetrator was 'just a kid' who 'needs counselling'.
'This highlights the urgent need for institutions to adopt a survivor-centric approach, free from bias and trivialisation of abuse,' the statement said.
The group urges the Malaysian public and policymakers to stay vigilant about the growing misuse of AI-generated content and its impact.
Immediate and firm action is needed to address digital safety and online gender-based violence, especially among minors.
In response to the issue, Women, Family and Community Development Minis­ter Datuk Seri Nancy Shukri said last week that a high-level brainstorming session to tackle the growing misuse of artificial intelligence (AI), particularly among school students on social media, is on the cards. The session will explore possible amendments to existing laws and strengthen enforcement mechanisms to keep pace with emerging digital threats.
Read more: Tackling AI misuse among students Exploitation of digital tools
Perpetrators can easily manipulate photos or videos to falsely portray individuals saying or doing things they never actually did. Photo: Freepik On April 8, the school confirmed that a male student had produced, sold and distributed AI-generated deepfake images of female schoolmates and alumni.
By April 15, 29 police reports were filed, with at least 38 victims identified, including very young children.
'This case underscores a deeply troubling trend: the exploitation of digital tools to perpetrate violence against minors. The ease with which deepfake technology can be weaponised – combined with weak oversight on platforms like Telegram and X – has made children increasingly vulnerable in online spaces.'
Deepfakes, as defined by the United Nations Population Fund, are a form of non-consensual image-based abuse.
Deepfake bots make it dangerously easy to manipulate photos or videos without the victim's consent. Perpetrators can easily manipulate photos or videos to falsely portray individuals saying or doing things they never actually did.
In the case of the school incident, one such deepfake image was viewed over 10,000 times on X, and a Telegram group with over 200 members continues to facilitate the circulation of these images, according to the statement.
'The multiplier effect of digital sharing amplifies trauma for survivors, most of whom are girls, and it severely impacts their mental health and future,' the group stressed.
While the perpetrator has been expelled and counselling services offered to survivors, the incident highlights the urgent need for Malaysia to strengthen its digital safety frameworks. Consistent enforcement necessary
Although the Online Safety Act 2024 marks progress, enforcement must be consistent and particularly vigilant in protecting minors from AI-generated abuse.
The coalition is advocating a three-pronged action plan: improving digital laws and platform regulation; enforcing clear policies on sexual misconduct in schools; and introducing Comprehensive Sexuality Education (CSE) for all.
'With Malaysia chairing Asean in 2025, there is an opportunity to lead by example. Countries like South Korea, Australia, and the European Union have introduced strong laws to combat deepfake-related crimes. Malaysia must criminalise the possession, creation, and distribution of non-consensual deepfakes, especially those involving children, and hold platforms accountable,' the group urged.
Teaching young people about consent, digital safety and respect from an early age is essential to dismantling the toxic cultures that enable such abuse. CSE must also extend to parents, teachers and caregivers to reinforce these values at home.
The statement was endorsed by Family Frontiers, Sisters in Islam (SiS), Justice for Sisters (JFS), Kolektif Feminis Malaysia, Association of Women Lawyers (AWL), Perak Women for Women Society (PWW), Women's Centre for Change Penang (WCC), Autism Inclusiveness Direct Action Group (Aida), Persatuan Kesedaran Komuniti Selangor (Empower), Sabah Women's Action Resource Group (Sawo), Sarawak Women for Women Society (SWWS), and Persatuan Sahabat Wanita Selangor (PSWS).
If you or someone you know is experiencing online gender-based violence and needs support, resources are available through KRYSS Network's OGBV Toolkit (www.kryss.network), Awam's Telenita Helpline (016-237 4221/016-228 4221), and WAO's Tina Helpline (018-988 8058).