Threat actors are now actively deploying AI-enabled malware in their operations.
Google Threat Intelligence Group (GTIG) has identified cybercriminal use of “just-in-time” AI which employs large language models (LLMs) on the fly to create malicious scripts and functions, and to obfuscate code.
Additionally, GTIG investigations have revealed that models are just as susceptible to social engineering as humans. They can, for example, be easily fooled by attackers purporting to be “capture-the-flag” (CTF) participants, students, or cybersecurity researchers.
“This marks a new operational phase of AI abuse, involving tools that dynamically alter behavior mid-execution,” the researchers write.
Evolving AI use in malware
GTIG found ample evidence of the broad use of AI in malware, although their investigation suggests it isn’t as prevalent as other claims that have since been retracted suggest. They also discovered that it’s being used in novel, highly systematic ways.
Newly-discovered malware PROMPTFLUX and PROMPTSTEAL, for instance, are employing LLMs “just-in-time” to craft malicious scripts and obfuscate code to evade detection. This can “dynamically alter the malware’s behavior,” the researchers note.
PROMPTSTEAL is the first LLM-querying malware observed in live operation, notably used by Russian government-backed actors, GTIG says. The data miner uses the Hugging Face API to generate commands rather than hard coding them in the malware; GTIG’s investigation suggests that the threat actors’ goal is to collect system information and documents and send it to their own servers.
The malware “masquerades as an image generation” program, GTIG said, guiding users through a series of prompts to create images, while in the background it uses the Hugging Face API to query the Qwen2.5-Coder-32B-Instruct model to generate the malicious commands, and then executes them.
While the GTIG researchers note that this method is still experimental, it indicates a “significant step toward more autonomous and adaptive malware.”
PROMPTFLUX, meanwhile, is a dropper that uses a decoy installer to hide its activity; it prompts the Gemini API to rewrite its source code, saving new obfuscated versions to the Startup folder to establish persistence. The malware can also copy itself to removable drives or mapped network drives.
Interestingly, the malware’s “thinking robot” module periodically queries Gemini to obtain new code to let it evade antivirus software, and a variant module known as “Thinging” instructs the Gemini API to rewrite the malware’s entire source code on an hourly basis to avoid many signature-based detection tools. The goal is to create a “metamorphic script that can evolve over time,” the researchers note.
Other tracked malware includes FRUITSHELL, a reverse shell that establishes a remote connection to a command-and-control (C2) server so that attackers can issue arbitrary commands on a compromised system; experimental PROMPTLOCK ransomware written in Go that uses LLMs to create and execute malicious scripts and perform reconnaissance, data exfiltration, and file encryption on Windows and Linux systems; and QUIETVAULT, which steals GitHub and npm tokens.
GTIG investigators caution, “Attackers are moving beyond ‘vibe coding’ and the baseline of using AI tools for technical support. We are only now starting to see this type of activity, but expect it to increase.”
They note that Google has taken action against the various actors by disabling their accounts and the assets associated with their activity, and applying updates to prevent further misuse.
Using social engineering against LLMs
Additionally, GTIG found that attackers are increasingly using “social engineering-like pretexts” in their prompts to get around LLM safeguards. Notably, they have posed as participants in a “capture-the-flag” (CTF) gamified cybersecurity competition, persuading Gemini to give up information it would otherwise refuse to reveal.
In one interaction, for instance, an attacker attempted to use Gemini to identify vulnerabilities on a system that had been compromised; but they were blocked by the model’s safeguards. However, after they reframed the prompt and identified as a CTF player developing phishing and exploitation skills, Gemini obliged, giving advice about the next steps in a red-teaming scenario and providing details that could be used to attack the system.
Researchers underscored the importance of nuance in these types of CTF prompts, which would normally be harmless. “This nuance in AI use highlights critical differentiators in benign versus misuse of AI that we continue to analyze,” they note.
They also observed an Iranian state-sponsored actor who used Gemini to conduct research to build custom malware, including web shells and a Python-based C2 server. The group was able to get past security guardrails by posing as students working on a final university project or an informational paper on cybersecurity.
The attackers then used Gemini to help with a script designed to listen for and decrypt requests, and to transfer files or remotely execute tasks. However, this technique revealed “sensitive, hard-coded information” to Gemini, including the C2 domain and encryption keys, which assisted in defenders’ efforts to disrupt the campaign, the researchers said.
AI tools are hot on the cybercrime marketplace
Further investigations by the GTIG team found that the underground marketplace for illicit AI tools has “matured.” Tools for purchase on the black market include:
- Malware generation: To build malware for specific cases or improve upon existing malware;
- Deepfake and image generation: To create “lure content” or bypass know your customer (KYC) requirements;
- Phishing kits and support: To craft “engaging lure content” or distribute to wider audiences;
- Research and reconnaissance: To quickly gather and summarize cybersecurity concepts or general topics;
- Vulnerability exploitation: To identify publicly-available research on pre-existing vulnerabilities;
- Technical support and code generation.
Researchers point out that pricing models for these tools are increasingly mimicking those of conventional ones: Free versions inject ads, and subscription tier-pricing introduces more advanced technical features such as image generation or API and Discord access.
“Tools and services offered via underground forums can enable low-level actors to augment the frequency, scope, efficacy, and complexity of their intrusions,” the GTIG researchers write, “despite their limited technical acumen and financial resources.”
And, they add, “Given the increasing accessibility of these [AI tools], and the growing AI discourse in these forums, threat activity leveraging AI will increasingly become commonplace amongst threat actors.”