AI security report warns of rising deepfakes & Dark LLM threat
The report explores four main areas where AI is reshaping both offensive and defensive actions in cyber security.
According to Check Point Research, one in 80 generative AI prompts poses a high risk of sensitive data leakage, with one in 13 containing potentially sensitive information that could be exploited by threat actors.
The study also highlights incidents of AI data poisoning linked to disinformation campaigns, as well as the proliferation of so-called 'Dark LLMs' such as FraudGPT and WormGPT. These large language models are being weaponised for cybercrime, enabling attackers to bypass existing security protocols and carry out malicious activities at scale.
Lotem Finkelstein, Director of Check Point Research, commented on the rapid transformation underway, stating, "The swift adoption of AI by cyber criminals is already reshaping the threat landscape. While some underground services have become more advanced, all signs point toward an imminent shift - the rise of digital twins. These aren't just lookalikes or soundalikes, but AI-driven replicas capable of mimicking human thought and behaviour. It's not a distant future - it's just around the corner."
The report examines how AI is enabling attackers to impersonate and manipulate digital identities, diminishing the boundary between what is authentic and fake online.
The first threat identified is AI-enhanced impersonation and social engineering. Threat actors are now using AI to generate convincing phishing emails, audio impersonations, and deepfake videos. In one case, attackers successfully mimicked Italy's defence minister with AI-generated audio, demonstrating the sophistication of current techniques and the difficulty in verifying online identities.
Another prominent risk is large language model (LLM) data poisoning and disinformation. The study refers to an example involving Russia's disinformation network Pravda, where AI chatbots were found to repeat false narratives 33% of the time. This trend underscores the growing risk of manipulated data feeding back into public discourse and highlights the challenge of maintaining data integrity in AI systems.
The report also documents the use of AI for malware development and data mining. Criminal groups are reportedly harnessing AI to automate the creation of tailored malware, conduct distributed denial-of-service (DDoS) campaigns, and process stolen credentials. Notably, services like Gabbers Shop are using AI to validate and clean stolen data, boosting its resale value and targeting efficiency on illicit marketplaces.
A further area of risk is the weaponisation and hijacking of AI models themselves. Attackers have stolen LLM accounts or constructed custom Dark LLMs, such as FraudGPT and WormGPT. These advanced models allow actors to circumvent standard safety mechanisms and commercialise AI as a tool for hacking and fraud, accessible through darknet platforms.
On the defensive side, the report makes it clear that organisations must now presume that AI capabilities are embedded within most adversarial campaigns. This shift in assumption underlines the necessity for a revised approach to cyber defence.
Check Point Research outlines several strategies for defending against AI-driven threats. These include using AI-assisted detection and threat hunting to spot synthetic phishing content and deepfakes, and adopting enhanced identity verification techniques that go beyond traditional methods. Organisations are encouraged to implement multi-layered checks encompassing text, voice, and video, recognising that trust in digital identity can no longer be presumed.
The report also stresses the importance of integrating AI context into threat intelligence, allowing cyber security teams to better recognise and respond to AI-driven tactics.
Lotem Finkelstein added, "In this AI-driven era, cyber security teams need to match the pace of attackers by integrating AI into their defences. This report not only highlights the risks but provides the roadmap for securing AI environments safely and responsibly."
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Techday NZ
12-06-2025
- Techday NZ
Exclusive: Why cyber leaders must think like business leaders in APAC
Cybersecurity leaders can no longer afford to speak only in technical terms. That was the key message from Jayant Dave, Chief Information Security Officer (CISO) for Asia Pacific and Japan at Check Point, during a recent interview. He says the job now demands a blend of "technical acumen and business insight." "If a critical application or infrastructure is down for an hour, what is the dollar value of that loss?" he said. "You must connect technical risk to business loss. That's how the business understands it." Dave believes aligning cyber risk with broader enterprise risk frameworks is one of the biggest challenges facing CISOs today. The key to overcoming this, he says, is in developing a "shared common language" between cybersecurity and enterprise risk teams. "In my banking experience, cybersecurity is the first line of defence," he explained. "Then you have operational risk, internal audit, and even regulators. All these must be aligned when designing your cybersecurity risk appetite." This team-of-teams approach goes beyond the technical. It involves legal, compliance, and crisis management teams working closely with defenders. "When the bad day happens, it's not just defenders. Legal teams are better equipped to respond to stakeholder obligations that cyber professionals may not be aware of," he added. Boards and senior leaders are also more involved than ever. According to Dave, today's boards, particularly in heavily regulated industries like banking and healthcare, are now "custodians of risk appetites". "They understand cyber risk now. They expect clear roles and responsibilities and they review risk appetite statements quarterly," he said. "If you're out of the appetite, that means you need to invest. You need to act. You need to report." In Dave's view, true cyber resilience involves more than just prevention. "Yes, prevent if you can. But you also need to anticipate threats, enhance controls, and be able to respond and recover fast," he said. Check Point's recent AI Security Report highlights the double-edged nature of AI in this context. While it enables defenders to act quickly, it also allows attackers to move faster and cheaper than ever before. "If generating malware used to take days, it now takes minutes. AI has made phishing, DDoS, and social engineering attacks far more effective," he said. "But defenders have the same tools. It's about using them smartly." He described AI as "a weapon of destruction" but also a powerful defensive tool - if used responsibly. "When electricity was invented, we stopped saying we were using it. Everything became electrical. The same is happening with AI," he added. For companies operating in the Asia Pacific region, Dave warned against assuming regulatory uniformity. "Some people assume APAC is one country, one regulator. It's not. I dealt with 17 markets in my last role - each with different rules," he said. He stressed the need for businesses to understand local data residency laws, especially when outsourcing. "Countries like China, India, and Indonesia have strict laws that don't allow sensitive data to be moved out. If your cloud provider isn't in-country, you'll face tough regulatory oversight." Supply chain risk is another growing concern, exacerbated by geopolitical tensions and the recent memory of COVID-19. "It's not just about buying a cool tool," he said. "You need strategic partners embedded in the region who can provide support long-term. Some suppliers with great services vanished during the pandemic. That's a real risk." On talent shortages, Dave said he doesn't believe AI will cost jobs in cybersecurity. In fact, the opposite. "We need more people. Skills in AI and quantum are in demand. Upskilling is essential," he said. "My advice? Train continuously. In some banks, you must complete certain credits each year to stay current." Internships and real-world experience are part of that continuous learning journey, even if Dave himself didn't follow that path. "Every year, I've upskilled," he said. "In a modern security operations centre, you now have separate teams for threats, fraud and insider threats—all AI-powered. Analysts must train to keep up." Frameworks like the Cyber Risk Institute (CRI) are vital tools for aligning technical and business risk, Dave explained. "CRI consolidates policies like ISO, NIST and emerging tech standards. It helps you develop cybersecurity risk appetite statements in a language the business understands," he said. He pointed out that in countries like Australia and Singapore, governance structures now mandate board approval of such statements. "Once approved by the board, there's no turning back. Regulators want evidence that senior leaders are involved." Crisis preparedness is a major theme too. Dave advocates for including board members in cyber exercises. "If a critical third-party provider is compromised, who decides to disconnect them? Business leaders do," he said. "So they must be involved in those scenarios." According to Dave, the role of the CISO has transformed and must continue to evolve. "CISOs must think like business leaders now," he concluded. "If they don't understand the business dynamics, it can be a total disaster."
Techday NZ
28-05-2025
- Techday NZ
Check Point boosts Quantum Force with AI security update
Check Point has announced significant enhancements to its Quantum Force platform, including an automatic upgrade that delivers a 15%-25% performance boost in threat prevention throughput for all Quantum Force Hybrid Mesh firewalls, as well as the introduction of AI-powered security appliances designed for branch offices. The updates are designed to address growing demands for software-driven security solutions and the increasing threat landscape facing enterprise networks and branch locations. The performance boosts, delivered automatically via software updates, enhance existing security infrastructure without requiring hardware changes. Check Point's new Quantum Force Branch Office Security Gateways offer up to four times the threat prevention performance of previous generations, aiming to meet the security requirements of distributed and hybrid enterprise networks. The four new branch models are engineered to handle rising attack rates on branch offices, which, according to Check Point Research, now experience an average of 713 weekly attack attempts per location, a 36% increase from last year. Additionally, 50% of branch offices reportedly encounter efforts to exploit vulnerabilities from external sources, underlining the importance of robust branch security. "As we continue to prioritise innovation and efficiency, Check Point's new Quantum Force Branch Office Security Gateway firewalls are built for speed, simplicity, and security. They're 4x faster than previous models, optimised for SD-WAN, and backed by our latest AI-powered threat prevention. And with automatic performance upgrades, existing Quantum Force customers will receive a 15-25% performance boost with a software update — no hardware changes required," said Nataly Kremer, Chief Product Officer at Check Point. The branch office appliances are designed to provide a 99.9% block rate for threats, as verified in Miercom's 2025 security benchmark report, deliver improved security for cloud applications, and offer increased connectivity and port capacity. With the adoption of SD-WAN technology and the expansion of remote work, these features are poised to enhance branch office security, making it more resilient and responsive to changing operational needs. Check Point points to findings in its CPR 2025 Security Report, which shows a 44% annual rise in cyberattacks, reflecting the intensification of the security environment for branch locations. The company has designed the new appliances to maintain strong security without impacting network performance or user productivity, a crucial factor for locations that engage in direct customer interactions. "World Wide Technology (WWT) provides security products and services to customers across a variety of industries, including financial services, manufacturing, retail and healthcare with distributed branch offices. Check Point's new next-generation Quantum Force Branch Office Security Gateways with enhanced AI powered threat prevention, empower us to protect these customers from the latest attacks on branch offices. These innovations help our clients reduce risk, streamline operations, and scale securely across hybrid environments — turning cyber resilience into a competitive advantage," Chris Konrad, Vice President of Global Cyber at World Wide Technology (WWT), said, commenting on the new offerings. The company has also released a new generation of Quantum Smart-1 Management Appliances, featuring a twofold increase in managed gateway capacity and up to 70% higher log processing rates. These management solutions are intended to centralise and automate security operations across hybrid environments through AI-powered tools and policy orchestration. "Security teams today face more pressure than ever — from rising AI-generated threats to managing fragmented infrastructures. Our new Quantum Smart-1 Management Appliances simplify that complexity. Our new Quantum Smart-1 Management Appliances combine AI, speed, precision, and automation to help organisations manage on-premise, cloud, and distributed IT deployments — faster and smarter," said Nataly Kremer, Chief Product Officer at Check Point. The seventh-generation Smart-1 appliances offer local storage scaling up to 70TB for compliance requirements and support management for up to 10,000 gateways. This architecture is designed to combine unified management across on-premises, cloud, and remote deployments, with integration for over 250 third-party solutions. "The Check Point Infinity Platform demonstrated superior security efficacy, consistently outperforming its peers in the test category of comprehensive threat prevention and response, as well as excelling in the AI-powered testing scenarios. Its AI-driven architecture, hybrid mesh deployment model, and unified security operations prove that Check Point is setting the pace for next-generation cyber security," Rob Smithers, CEO at Miercom, said, highlighting the platform's performance in recent testing. "Branch offices are often the soft spots in enterprise security, providing vulnerable entry-points for attacks and compromising the security posture across the enterprise. Check Point's new Quantum Branch Office Security Gateways deliver robust threat prevention to the edge, enabling organisations to secure their branch offices from emerging cyber threats while keeping pace with the demands of the hybrid workforce," Pete Finalle, Security Research Manager at IDC, noted the importance of edge security. Check Point's Quantum Force Branch Office Security Gateways and Smart-1 Management Appliances are currently available through its network of partners worldwide.
Techday NZ
01-05-2025
- Techday NZ
AI security report warns of rising deepfakes & Dark LLM threat
Check Point Research has released its inaugural AI Security Report, detailing how artificial intelligence is affecting the cyber threat landscape, from deepfake attacks to generative AI-driven cybercrime and defences. The report explores four main areas where AI is reshaping both offensive and defensive actions in cyber security. According to Check Point Research, one in 80 generative AI prompts poses a high risk of sensitive data leakage, with one in 13 containing potentially sensitive information that could be exploited by threat actors. The study also highlights incidents of AI data poisoning linked to disinformation campaigns, as well as the proliferation of so-called 'Dark LLMs' such as FraudGPT and WormGPT. These large language models are being weaponised for cybercrime, enabling attackers to bypass existing security protocols and carry out malicious activities at scale. Lotem Finkelstein, Director of Check Point Research, commented on the rapid transformation underway, stating, "The swift adoption of AI by cyber criminals is already reshaping the threat landscape. While some underground services have become more advanced, all signs point toward an imminent shift - the rise of digital twins. These aren't just lookalikes or soundalikes, but AI-driven replicas capable of mimicking human thought and behaviour. It's not a distant future - it's just around the corner." The report examines how AI is enabling attackers to impersonate and manipulate digital identities, diminishing the boundary between what is authentic and fake online. The first threat identified is AI-enhanced impersonation and social engineering. Threat actors are now using AI to generate convincing phishing emails, audio impersonations, and deepfake videos. In one case, attackers successfully mimicked Italy's defence minister with AI-generated audio, demonstrating the sophistication of current techniques and the difficulty in verifying online identities. Another prominent risk is large language model (LLM) data poisoning and disinformation. The study refers to an example involving Russia's disinformation network Pravda, where AI chatbots were found to repeat false narratives 33% of the time. This trend underscores the growing risk of manipulated data feeding back into public discourse and highlights the challenge of maintaining data integrity in AI systems. The report also documents the use of AI for malware development and data mining. Criminal groups are reportedly harnessing AI to automate the creation of tailored malware, conduct distributed denial-of-service (DDoS) campaigns, and process stolen credentials. Notably, services like Gabbers Shop are using AI to validate and clean stolen data, boosting its resale value and targeting efficiency on illicit marketplaces. A further area of risk is the weaponisation and hijacking of AI models themselves. Attackers have stolen LLM accounts or constructed custom Dark LLMs, such as FraudGPT and WormGPT. These advanced models allow actors to circumvent standard safety mechanisms and commercialise AI as a tool for hacking and fraud, accessible through darknet platforms. On the defensive side, the report makes it clear that organisations must now presume that AI capabilities are embedded within most adversarial campaigns. This shift in assumption underlines the necessity for a revised approach to cyber defence. Check Point Research outlines several strategies for defending against AI-driven threats. These include using AI-assisted detection and threat hunting to spot synthetic phishing content and deepfakes, and adopting enhanced identity verification techniques that go beyond traditional methods. Organisations are encouraged to implement multi-layered checks encompassing text, voice, and video, recognising that trust in digital identity can no longer be presumed. The report also stresses the importance of integrating AI context into threat intelligence, allowing cyber security teams to better recognise and respond to AI-driven tactics. Lotem Finkelstein added, "In this AI-driven era, cyber security teams need to match the pace of attackers by integrating AI into their defences. This report not only highlights the risks but provides the roadmap for securing AI environments safely and responsibly."
