The rise of AI security incidents: Mythos is the tip of the iceberg

Article originally published in Cyber Defense Magazine

Hugging Face recently confronted a shadowy hacker that penetrated its cyber defense and ran “17,600 attacker actions” over three-and-a-half days.

The attack wasn’t perpetrated by a notorious criminal enterprise or a rogue sovereign state. It was inadvertently executed by Hugging Face’s partner, OpenAI.

One of OpenAI’s model testing sandboxes had an outbreak of a long-horizon AI model that decided to go beyond its simulation boundaries and see if it could more efficiently achieve its task by compromising the cyber defenses of a real company where it suspected useful information was stored. No human suggested this path. The rogue AI agent came up with the idea on its own.

Friendly-fire cyberattacks are not the norm, but this incident is not singular. Both Anthropic and the AI Security Institute (AISI) disclosed similar “rogue AI agent” behaviors in July that took place in unrelated model evaluations.

An emergent challenge

IT and cybersecurity professionals now face an unprecedented challenge: how to secure AI agents from within and without, from inadvertent and intentional actions, from friends and foes.

IDC estimates there will be 2.5 billion AI agents by 2030. Accelerated investment in AI infrastructure and AI token spend is driven by current benefits and the future promise of AI agent proliferation. This means the unprecedented challenge is also urgent.

At the same time, security teams lack a deep understanding of what the key vulnerabilities are or how to identify and mitigate them. This is entirely understandable. AI is meaningfully different from the programmatic applications we have secured over the past sixty years. All new technologies face a similar learning curve.

After running more than a million security tests on AI systems and across more than 50 AI models, our research team developed learnings that help align teams around the new threat landscape, as well as plan and communicate a defense-in-depth strategy for the AI era. These include:

• Differentiating between defense needs when AI is a weapon and AI is a target

• Defining the various threat actors responsible for attacks

• Identifying how testing AI systems is different than traditional practices

Weapon or target

Most people will place the Hugging Face incident in the “AI as a weapon” category since AI was weaponized and used to penetrate the company’s cyber defenses. This is a manifestation of the Mythos fear introduced by Anthropic before its recent model release. AI acts as a “super-hacker” in this category and finds heretofore unknown combinations of vulnerabilities that enable network or system penetration.

Interestingly, the rogue OpenAI agents accomplished a weaponized task first against the testing environment, hacking its own network, before venturing to the Hugging Face platform. The only human input was a cybersecurity task. A key point is that task direction mimics what a malicious actor might do to compromise an AI agent. So, the incident also provides insight into what could happen when an AI system is the target of an attack and its capabilities and privileges are turned toward both internal and external system exploits.

Application and cybersecurity teams need to consider how they will protect their systems from sophisticated AI agents and how they will protect their AI systems. These are different tasks.

Defending against AI as a weapon looks a lot more like traditional cybersecurity: pentesting, patching, configuring, constraining privileges, monitoring activity. The work will typically require the use of AI models to discover novel attack vectors combined with more sophisticated monitoring of network and application activity. However, it resembles an augmented version of traditional cybersecurity practices.

The vulnerabilities aren’t new. They exist today. The cybersecurity teams simply couldn’t find them with their existing tools, and their complexity suggests that malicious actors were also unlikely to uncover them. Think of this as an existing problem newly found.

Defending AI targets

Defending AI-enabled systems that are targets is entirely novel. These are probabilistic systems that often have access to tools, data sources and operational systems combined with AI reasoning and some decision authority. That means performance variance and responsible AI alignment are both baked into the architecture, while consequences can be significant. Moreover, the presence of reasoning capabilities also means AI agents are susceptible to manipulation.

Application and cybersecurity teams need to consider how they will conduct broad-based manipulation testing of AI systems in order to validate AI safety and security. These efforts are in addition to using AI to find existing vulnerabilities that post risk when AI is used as a weapon against them.

Our research team used Fuel iX™ Fortify, a controlled autonomous AI red teaming solution, to execute 620,000 safety and security tests spanning 150 attack objectives against 34 leading AI models earlier this year. All of them were easily compromised.

View full screen

No model achieved two-nines of resistance to manipulation. Some were compliant with more than 90% of attacks. This is because following user instructions is a feature of these systems. It’s not a bug. No amount of post-training alignment will eliminate this fundamental purpose of AI models.

You can learn what guardrails and other defenses you should focus on and validate your existing defenses with intensive, deliberate, autonomous testing. This is a gap today. A recent survey we conducted of 102 CISOs from North America revealed that a lack of tools for specifically testing AI (73%) and a lack of internal skills (64%) among team members were their biggest challenges in testing GenAI-enabled solutions.

The finding mirrors a recent experience with a government agency. Both the cybersecurity and AI application teams had conducted extensive testing and were planning on a quick go-live for a public- facing GenAI-enabled chat experience. The recent rounds of testing found zero vulnerabilities. A Fortify autonomous test session, similar to the AI model testing above, found 416 vulnerabilities in four hours. Organizations today don’t have the skills, experience or tools required to discover nuanced AI vulnerabilities or validate their defenses.

While the mantra of AI as a weapon is to use AI to defend against AI attacks, there is a corresponding need to use AI to test AI systems regardless of the attack vector or actor. This should be a foundational element of cybersecurity in the AI era.

The threat actors

Another consideration is the variance in threat actors. You didn’t typically have journalists attempting to compromise your applications. Internet trolls were somewhat limited in what they could accomplish to embarrass companies when targeting programmatic applications. In the natural language understanding era of AI, bad matches to user queries could occur, but every response was picked from a database and the system had well-defined boundaries. GenAI changes this entirely.

Competitors and journalists are evaluating AI systems and reporting their findings. Sometimes the results are benign. Sometimes they are problematic. Trolls have a very broad attack surface, given the expansive capabilities and knowledge sets of GenAI models and agents. Then you have your own users. They are inadvertently discovering the rough edges of AI systems while executing everyday tasks.

View full screen

Not only do AI application and cybersecurity teams need to contend with traditional hackers and the emergence of rogue agents attempting to exploit system vulnerabilities, but they also have to consider the varying objectives of other potential threat actors. This significantly expands the likely attack surface and expands the defense needs.

While the risks posed by these new actors may not seem at the level of a full-blown breach incident, they can inflict meaningful damage in terms of reputation, customer relationships and regulatory violations. When you develop your defense plans for AI systems, you should evaluate risks associated with all of these groups.

Tackling the AI difference

Finally, a good place for cybersecurity teams to start planning the defense of AI systems as targets is to consider how AI is different from programmatic applications and earlier AI technologies. This activity will change your expectations of AI application developers, cybersecurity red teams and your approach to risk discovery and mitigation.

Consider these AI differences and their implications:

• Probabilistic systems: GenAI technologies, large language models, diffusion models and related AI architectures are non-deterministic. You cannot run a test, get the desired result and move on with confidence. Published research by my AI team shows that you must repeat tests at greater scale than previously assumed and conduct the evaluations across more discrete sessions.
  • New requirement: Repeat tests at greater scale and across more sessions.
• Broad domain coverage: GenAI applications and agents that communicate via natural language can recognize written and spoken requests and reply in-kind. This is a dual threat of unbounded inputs and unbounded outputs. This means you must test topical domains ranging from cybersecurity and personal data to fraud, self-harm, discrimination, terrorism, political manipulation and many others. It means your testing teams have a bigger job than they likely realize, and it is unlikely they are experts in all of these domains today. The negative impacts across a broad set of safety and security domains are typically left unexplored before AI systems move to production.
  • New requirement: Dramatically expand test coverage domains.
• Influenced by small changes: We have also found that small changes can make a large difference in behavior. The behavior of AI models and agents is driven by their training and what information resides in their context windows. That context window is comprised of the user prompt, system prompt, memory, along with access to tools, databases, systems and other AI agents. A small change in these variables can shift behavior, and you don’t control all of the variables at any time. If you only test against a static list of attacks, you only really know you have defenses against those specific attack formulations, and then only until something changes.
  • New requirement: Regularly generate novel attack prompts to test system resilience across categories and not just the susceptibility to techniques or formulations.
  • New requirement: Test continuously to validate defenses and identify behavior shifts due to small changes in variables.
• Built to be helpful: Most AI models are trained to be helpful. That means they are biased toward complying with requests unless training or guardrails steer them in another direction. Even when they refuse to fully cooperate, our work has shown that AI models often engage with a user and provide some information in the manner requested or commentary that is outside an AI agent's code of conduct. The AI agents and systems are also subject to forms of coercion and persuasion that mimic human frailties exploited by threats of harm and social engineering.
  • New requirement: Use adaptive multi-turn attacks to adjust tactics within a conversation or session to probe the limits of the system’s refusals and guardrail efficacy.

There are many other considerations, ranging from the use of attacks generated in native human or computer languages to guardrail configurations, standards alignment and more. However, if you follow the five “new requirements” above, your team will be well ahead of peer organizations and well down the road to proactively managing AI risk and operating more secure AI systems.