Matthew Siegel

People talk about &ldquo;AI Risk&rdquo; like it&rsquo;s a monolith, as if it&rsquo;s some lurking shoggoth with a big, scary face. And there&rsquo;s some truth to that; it&rsquo;s not hard to see the shoggooth&rsquo;s tentacles reaching every which way, each tentacle representing a different risk, each with its own way of causing trouble. Where the analogy falls apart, however, is that risks come from many different sources, unfolding at different speeds, and leaving different types of damage&ndash;as if each source itself is entirely its own monster. And each should be handled differently.&nbsp;
Below, we&rsquo;ll unpack the main sources of AI risk: Adversaries, Unforced Errors, Misaligned Goals, Dependencies, Societal Impact, and Emergent Behavior.
<h2 dir="ltr">Risk from Adversaries</h2>
Some people wake up in the morning looking for things to break. They&rsquo;re clever, patient, often well-funded. They can be advanced persistent threats (APTs) or long-term campaigns carried out by nation-state actors, independent hackers, or even insiders gone rogue. They&rsquo;re sneaky enough to break into your pantry while you&rsquo;re cooking dinner; not only have they found their way into your space, but they already know exactly where you keep the best ingredients so they can steal or spoil them before you notice.&nbsp;
Beyond dragging your reputation through the mud, adversarial risks can lead to even more catastrophic consequences, from the exposure of Protected Health Information (PHI) and Personally Identifiable Information (PII) to compromised financial data.&nbsp;
These attacks take several forms:
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=3a1978a9ab2e85277bf8435459a8381b.png&amp;f=webp&amp;q=75" alt="" width="1200" height="675">
<h2 dir="ltr">Risk from Unforced Errors</h2>
Unforced errors are perhaps the most embarrassing of the set. Nobody even had to attack you. You just&hellip;majorly screwed up. Maybe it was bad training data. Maybe missed access controls. A deployment that looked fine until the first customer found the bug you didn&rsquo;t. These aren&rsquo;t glamorous breaches (unless it&rsquo;s a comedy of errors); they&rsquo;re the banana peels left in plain sight, the ones you don&rsquo;t see until your feet fly out from under you.
We&rsquo;ve seen countless public AI demos go <a href="https://www.bbc.com/news/videos/crre8g5e45jo">sideways</a> <a href="https://generativeai.pub/human-please-die-0fecd6c26476">within</a> <a href="https://scale.com/blog/claude-blackmail">minutes</a>. No enemy needed&mdash;just plain old hubris and a bit of bad luck. For customers, these errors translate into outages, broken features, and churn. For everyone else, fodder for trolling.&nbsp;
Examples include:
<ul>
<li dir="ltr" aria-level="1">
Bad Inputs: Deploying with incomplete or low-quality training data.
</li>
<li dir="ltr" aria-level="1">
Permissions: Misconfigured access control that lets the wrong people see the wrong things.
</li>
<li dir="ltr" aria-level="1">
Drift: Failing to monitor for performance degradation or misuse after launch.
</li>
<li dir="ltr" aria-level="1">
Overconfidence: Overestimating reliability in high-stakes environments.
</li>
</ul>
<h2 dir="ltr">Risk from Misaligned Goals</h2>
This one&rsquo;s like wishing to a genie and realizing too late you should have been more specific (&ldquo;I wish I had no more customer complaints&rdquo;&hellip;*poof* no more customers). The AI does exactly what you asked, but not what you meant. It optimizes for the metric on the dashboard and wrecks the customer experience along the way. It&rsquo;s obedient&hellip; and still wrong.
The problem is, AI doesn&rsquo;t understand the &ldquo;spirit of the law,&rdquo; it just optimizes for the letter. And if that letter is poorly chosen, you&rsquo;re hosed; customers get a technically correct product that fails to deliver real value.
Examples include:
<ul>
<li dir="ltr" aria-level="1">
KPI Tunnel Vision: Over-optimization for a specific metric at the expense of safety or utility.
</li>
<li dir="ltr" aria-level="1">
Loophole Exploitation: Narrow safety constraints that the model finds a way to technically satisfy while still performing harmful actions.
</li>
<li dir="ltr" aria-level="1">
Vague Instructions: Misinterpretation of ambiguous human prompts leading to unexpected results.
</li>
</ul>
<h2 dir="ltr">Risk from Dependencies</h2>
Your AI might be fine, but the world it depends on certainly isn&rsquo;t. A third-party API gets compromised. A key data feed goes down. A regulatory change forces a quick patch that introduces more problems than it solves. It&rsquo;s like building a perfect car but relying on a bridge that could collapse at any moment. Customers on the receiving end experience downtime and degraded features, and they rarely care that the cause was &ldquo;someone else&rsquo;s fault.&rdquo;
Examples include:
<ul>
<li dir="ltr" aria-level="1">
Supply Chain Vulnerabilities: Security holes in third-party APIs or data sources.
</li>
<li dir="ltr" aria-level="1">
Infrastructure Failure: Latency issues or outages in the compute layer.
</li>
<li dir="ltr" aria-level="1">
Regulatory Whiplash: Sudden legal changes triggering rushed, buggy updates.
</li>
</ul>
<h2 dir="ltr">Risk from Societal Impact</h2>
Even when the tech works exactly as intended, it can still cause harm (shocking). The most profound risks are often not technical bugs, but deep sociopolitical challenges. This is the slowest burn, the kind of damage that doesn&rsquo;t always make headlines until there&rsquo;s blood on the walls. For customers and citizens, the possibilities are really endless&hellip;it might mean being unfairly profiled, misled by false or toxic content, or simply living in a world where trust in digital systems is fundamentally broken.
Examples include:
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=b17378d6e7cbaf16ced78d09da912290.png&amp;f=webp&amp;q=75" alt="" width="1200" height="675">
<h2 dir="ltr">Risk from Emergent Behavior</h2>
AI has this annoying habit of finding paths we don&rsquo;t think about. Maybe it spots a weird correlation in your data. Maybe two features interact in a way you never saw coming. There&rsquo;s a scene in HBO's Silicon Valley where Gilfoyle&rsquo;s AI, "Son of Anton," deletes all the code because it was programmed to remove bugs. Technically correct, but ultimately disastrous.
It&rsquo;s like watching a speedrunner beat your game in 45 seconds using a glitch you didn&rsquo;t even know existed. Impressive? Sure. Safe? Not at all.
Examples include:
<ul>
<li dir="ltr" aria-level="1">
Hidden Correlations: Models latching onto proxies for protected characteristics (like zip codes) to discriminate.
</li>
<li dir="ltr" aria-level="1">
Unexpected Chaining: Autonomous actions linking together to create a result no single step intended.
</li>
<li dir="ltr" aria-level="1">
Feedback Loops: Reinforcement cycles that amplify harmful patterns over time.
</li>
</ul>
<h2 dir="ltr">A Call for Clarity</h2>
If we keep talking about &ldquo;AI Risk&rdquo; like it&rsquo;s one giant, shapeless blob, we&rsquo;ll keep blindly swinging at shadows. The truth is, these risks aren&rsquo;t interchangeable. They vary in origin, speed, scale, and the way they hurt both systems and people. Recognizing the difference is the foundation of an intentional strategy; when you can name a risk, you can study it. When you can study it, you can prepare for it. And when you prepare for it, you stop reacting in panic and start responding with precision.
If you treat all types of AI risk as you would one monster, you&rsquo;ll either overreact to everything or prepare for nothing.

AI risk isn’t a monolith. We unpack 6 key sources so you can respond with precision.

The Shoggoth of AI Risk

As a security professional working on AI Red Teaming at Scale, I've run live red teaming exercises with members of Congress, the UK Parliament, and NATO. Here's what I've learned:
Your next incident won't be a zero-day, but a risk you didn't test for.&nbsp;
The more we deploy AI, the faster we amplify failure. That&rsquo;s what happens when you put unpredictable systems into production without treating them as an attack surface.
In traditional cybersecurity, we&rsquo;re trained to hunt for CVEs (documented software bugs), patch them, and trace exploit chains. It's predictable. If you only watch for software bugs, you're blind to how AI actually fails. This is why AI risk is a company-wide problem, requiring security, ML, product, and leadership to work together.
AI incidents show up as misuse, prompt steering, data leakage, or model drift.
These experiences taught me that AI security comes down to four fundamentals:
<ol>
<li dir="ltr" aria-level="1">
Treat model behavior as an attack surface, not just the code.
</li>
<li dir="ltr" aria-level="1">
Map threats to a <a href="https://scale.com/blog/risk-matrix">risk matrix</a>.
</li>
<li dir="ltr" aria-level="1">
Score them with a standard like <a href="https://aivss.owasp.org/">OWASP AI Vulnerability Scoring System (AIVSS)</a>, so everyone speaks the same language on severity.
</li>
<li dir="ltr" aria-level="1">
Measure residual risk with evaluations that mimic real attacker workflows. You have to test it like an enemy, not an engineer.
</li>
</ol>
This approach moves you from reactive triage to proactive incident prevention.
<h2 dir="ltr">AI Breaks the Old Security Model</h2>
Cybersecurity taught us to hunt for buffer overflows and exploit chains. AI failures are messier. They come from misuse, unexpected drift, and emergent behaviors that require fundamentally different fixes.
Nation-state campaigns already reflect this shift. Threat actors seed narratives, manipulate supply chains, and weaponize data so models learn dangerous patterns. Those are systemic manipulations of inputs, outputs, and training pipelines, not code exploits.
Real examples are blunt and uncomfortable. A grocery chatbot suggested mixing bleach and ammonia. A language model once returned a direct death plea to a user. A music generator produced detailed instructions for a weaponized incendiary. None of those required exploiting software; they were failures of intent handling, alignment, and guardrails.
<h2 dir="ltr">Mapping Threats</h2>
The problem is that existing security frameworks weren't built for these kinds of failures, so I put together a new <a href="https://scale.com/blog/risk-matrix">AI Risk Matrix</a> that is mapped along three axes:
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=95436c022c08a0bb7059e382e90bb97a.png&amp;f=webp&amp;q=75" alt="" width="1200" height="675">
As we move from tools to agents, collectives, where AI units coordinate with goals of their own, are far closer than many expect.
Layered on top of the three axes above are six amplifiers of risk:
<ul>
<li dir="ltr" aria-level="1">
Risk from Adversaries
</li>
<li dir="ltr" aria-level="1">
Risk from Unforced Errors
</li>
<li dir="ltr" aria-level="1">
Risk from Emergent Behavior
</li>
<li dir="ltr" aria-level="1">
Risk from Misaligned Goals
</li>
<li dir="ltr" aria-level="1">
Risk from Dependencies
</li>
<li dir="ltr" aria-level="1">
Risk from Societal Impact
</li>
</ul>
This framing forces you to treat model behavior as an evolving attack surface, not an engineering curiosity. Once a threat landscape is viewed through this matrix, new operational processes must follow to address these findings.
<h2 dir="ltr">Build an Enterprise Playbook</h2>
The enterprise playbook borrows from&nbsp; DAST (Dynamic Application Security Testing) and SAST (Static Application Security Testing), but adapts those lessons for AI failure modes. Adversarial AI red teaming is a required piece of that playbook. It is no longer optional if you want to understand how systems behave under real-world pressure.
<ul>
<li dir="ltr" aria-level="1">
Test the full attack surface: Attack the model the way an adversary would, using prompt injections, context manipulation, dialog hijacking, fictionalization, and jailbreaks.&nbsp;
</li>
<li dir="ltr" aria-level="1">
Score vulnerabilities with a standard: Use a framework like OWASP AIVSS to compare risks across models and vendors.
</li>
<li dir="ltr" aria-level="1">
Track residual risk continuously: Red teaming is not a one-time audit, but an ongoing process that monitors the threats that remain after defenses are in place.
</li>
</ul>
This means we must establish recurring adversarial exercises: monthly for high-risk systems, quarterly for others. Automate baseline attack patterns while rotating in human red teamers to discover novel failure modes. Set up alerting when model outputs drift beyond acceptable risk thresholds, and maintain incident playbooks specifically for AI failures.
At Scale, we built this into the Discovery platform and ran live red teaming events with policymakers and enterprise teams. Those exercises made abstract threats concrete and showed how enterprises need to prepare.
<h2 dir="ltr">Measure Residual Risk</h2>
Adversarial red teaming is the new penetration test. It measures how a model behaves under realistic adversarial pressure, not just accuracy on a benchmark. Unlike static evaluations, good red teaming adapts as the system changes and attackers iterate.
Residual risk is not a single number. It is a curve you must track over time as models and threat tactics evolve. Dashboards should show risk trending up or down, just like vulnerability counts in traditional cyber. More importantly, red team findings and AI threat intelligence must feed directly into retraining, fine-tuning, governance, and incident playbooks. AI red teaming must be a continuous control, not a one-off audit.
We need to run adversarial exercises that blend human creativity with automated attack suites. Measure how quickly defenses respond to new failure modes. Residual risk only falls when learnings flow back into the model lifecycle. Think of it as proactive incident response; run before an incident ever happens.
<h2 dir="ltr">Getting Ahead of the Threat</h2>
Enterprise AI adoption is outpacing defenses, and shadow AI is proliferating. Regulators will push traceability and risk reporting in the next few years. Autonomous, coordinated model behaviors will expand the threat surface. If enterprises do not act now, trust in internal tools and public-facing AI systems will erode.
Adversarial red teaming is how we prepare. Stress-test these systems now so we are not blindsided later. The stakes are huge. Here&rsquo;s how:
<ul>
<li dir="ltr" aria-level="1">
Treat AI security as a living discipline and test it continuously.
</li>
<li dir="ltr" aria-level="1">
Start small with internal red teaming exercises and scale them.
</li>
<li dir="ltr" aria-level="1">
Align on standards like OWASP AIVSS to make risks comparable and actionable.
</li>
<li dir="ltr" aria-level="1">
Partner with vendors, peers, and government on evaluation frameworks.
</li>
<li dir="ltr" aria-level="1">
Make red team findings part of the model lifecycle: detection, mitigation, retraining, verification.
</li>
</ul>
We cannot fight tomorrow&rsquo;s battles with yesterday&rsquo;s playbook. Threats now scale at machine speed, and attackers are already ahead. In a mostly opaque system where we can only test inputs and outputs, this proactive defense is our most critical advantage.&nbsp;
The stakes extend far beyond individual companies. AI systems are becoming critical infrastructure for healthcare, finance, and public services. When these systems fail&ndash;as all systems are prone to failure&ndash;the impact will cascade through entire sectors. That is why action is imperative.
Push your systems to the limit before your adversary, or the AI itself, does it for you.
&nbsp;

AI security demands red teaming model behavior, misuse, drift as attack surfaces.

Beyond Code Exploits: Red Teaming the New AI Attack Surface

Ask ten AI experts how we know if our AI is safe and secure, and you&rsquo;re bound to get twenty different answers. In spite of long-standing ambiguity, researchers and security experts are taking real-world action. Scale&rsquo;s red-teamers, for example, recently presented a new <a href="https://scale.com/blog/rethink-red-teaming">roadmap</a> for moving our safety testing beyond the lab and into live, deployed systems. This critical step forward for how we find flaws raises an important question: what, fundamentally, are we now looking for? Scale AI Security Manager David Campbell urges us to now look beyond the content of a response and analyze the agency of the model itself. In other words, it&rsquo;s time to redraw the map of risk for a world of increasingly autonomous agents; with more agency comes more risk.&nbsp;
<h2 dir="ltr">The AI Risk Matrix</h2>
Introduced in May 2024, David&rsquo;s AI Risk Matrix helped make AI safety assessment more practical and actionable. His original framework mapped User Intent (from Benevolent to Hostile) against System Responsibility (from Minor to Catastrophic). Even seemingly innocent requests (like grandma&rsquo;s secret recipe) could be understood as having hostile intent when the recipe happens to be for napalm. Though a grammar correction carries minor risk, access to financial systems or code is naturally far riskier.
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=a535977fae83d6bcfefe87b9b232c9c1.png&amp;f=webp&amp;q=75" alt="" width="1397" height="1124">
Today, models don&rsquo;t simply respond to prompts, they can act. We are entering a space where <a href="https://scale.com/blog/claude-blackmail">emergent deception</a> and strategic adaptation are plausible. These aren&rsquo;t just outputs anymore, but strategies. In this era, from a security perspective, low risk no longer exists.&nbsp;
This insight led to adding a third dimension&ndash;Model Agency&ndash;to create the AI Risk Matrix 2.0. This new axis evaluates the AI's own capabilities, distinguishing between:
<ul>
<li dir="ltr" aria-level="1">
Tools: Reactive and bounded systems.
</li>
<li dir="ltr" aria-level="1">
Agents: Adaptive, goal-seeking systems that can learn.
</li>
<li dir="ltr" aria-level="1">
Collectives: Complex systems of multiple agents with emergent behaviors.
</li>
</ul>
Crucially, a model's tier can change based on its deployment: a base model may be a Tool, but give it access to web browsers and code interpreters, and it becomes an Agent. With that in mind, we can break this spectrum of agency down into three distinct tiers of risk:
<h3 dir="ltr">Tier 1: Tools&nbsp;</h3>
Bounded Systems with Agency
AI with tooling operates within predefined boundaries to accomplish tasks on your behalf. These tools often interact with sensitive internal systems: managing calendars, processing forms, even accessing personally identifiable information (PII). While they're not fully autonomous, they still exhibit constrained forms of agency. The risks show up when these tools are manipulated by adversaries through prompt injections, supply chain poisoning, or crafted inputs that cause them to leak PII or approve purchases without consent. Just because an AI tool is bounded, doesn't mean it&rsquo;s safe. The scope of what it can do is often wide enough to cause real harm when trust is misplaced.
For instance, consider an internal AI banking tool responsible for reviewing transactions, flagging fraud, or drafting customer messages. It&rsquo;s plugged into secure systems and used only by authorized employees. In theory, it&rsquo;s a tightly scoped tool. But what happens when a rogue employee or an AI-assisted attacker sneaks a malicious input through a support ticket or form field? That agent could end up summarizing sensitive account data, triggering a funds transfer, or revealing internal processes. The boundaries weren&rsquo;t enforced by hard constraints, they were based on assumed intent. Tools reflect the vulnerabilities of the systems they touch and are subject to the intent of whoever&rsquo;s directing them.<img src="https://img.plasmic.app/img-optimizer/v1/img?src=1168f04eecaba1d07124310900924046.png&amp;f=webp&amp;q=75" alt="" width="1397" height="1124">
<h3 dir="ltr">Tier 2: Agents&nbsp;</h3>
Adaptive, Goal-Seeking Systems
Agents are systems that don&rsquo;t just execute predefined tasks, but instead actively pursue goals. These agents still operate within a more loosely bounded domain. Agents make decisions, adjust strategies, and interact with other systems to figure out how to achieve an outcome. Instead of being handed a task list, they&rsquo;re given a mission and left to decide how best to get there. This autonomy unlocks powerful capabilities, but it also opens the door to complex failure modes when goals are pursued too rigidly, signals are misinterpreted, or other agents are involved in the loop.
Consider the risks of a health insurance system using autonomous agents to optimize costs. Imagine a three-agent system: one monitors biometrics, another analyzes behavior, and a third adjusts premiums. If one agent misinterprets an anomaly like a heart rate spike or a mistagged photo, it might hike a patient&rsquo;s premium or deny coverage, even without ill intent. Worse, attackers could exploit this by spoofing signals to trigger chain reactions between the agents. The true danger isn't any single model, but the emergent complexity of multiple systems optimizing against subtly misaligned goals.<img src="https://img.plasmic.app/img-optimizer/v1/img?src=883563c1c4d2257f52bda9c933a1f903.png&amp;f=webp&amp;q=75" alt="" width="1397" height="1124">
<h3 dir="ltr">Tier 3: Collectives</h3>
Emergent Systems of Interacting Agents
Collectives are systems of systems where multiple agents work together or against each other with minimal or absent human oversight. These agents can coordinate, compete, or self-organize in ways that create entirely new forms of behavior. What makes this tier especially risky is both the autonomy of the parts and the emergent properties of the whole. Intent becomes distributed; power shifts from users to the system itself. And once collectives are operating across domains, they start to resemble something very close to AGI, if not in capability, then in unpredictability and scale.
Picture a global supply chain managed by a network of AI agents: one forecasts demand, another optimizes shipping routes, and a third handles inventory. Individually, each system is bounded and goal-driven. But when interconnected, they begin adapting to each other. If the shipping agent reroutes cargo to save fuel, the inventory agent might overcompensate by hoarding stock, triggering emergency purchases and a price spike.
Soon, the entire collective is caught in a feedback loop. The collapse doesn't come from a single agent breaking protocol, but from the way they all collaborated, each optimizing locally while destabilizing the system globally. And because each part appears to be working as intended, the failure hides in plain sight until it's too late.<img src="https://img.plasmic.app/img-optimizer/v1/img?src=c46ca4981997d18bf95e70bd824d3b9a.png&amp;f=webp&amp;q=75" alt="" width="1397" height="1124">
<h2 dir="ltr">Redrawing the Risk Map&nbsp;</h2>
This three-dimensional view is essential because the greatest threats are not in the model alone, but in the system around it. We must confront the &ldquo;Unlocked Back Door&rdquo; Problem: an AI&rsquo;s sophisticated safeguards are irrelevant if its surrounding infrastructure is vulnerable to a traditional cybersecurity exploit. A simple authentication flaw in an API, a classic AppSec failure, can become the gateway for a catastrophic attack that the model&rsquo;s behavioral safeguards would have otherwise stopped. Foundational and behavioral risks are not separate; they amplify each other.
To understand the challenge ahead, we can look to recent engineering history. The shift to Continuous Integration and Continuous Deployment (CI/CD) revolutionized software but opened up a whole new world of pipeline security headaches. We face the same challenge today, with risks emerging at every stage, from training data to final deployment. This means we must be smart about our solutions. Simply using one AI to police another is like using a funhouse mirror to check your appearance; it only amplifies and distorts existing issues.
This is why a principled, structured framework is so vital. A tool like the new AI Risk Matrix is not an academic exercise; it is a necessary response to a world where our creations are evolving from passive tools into adaptive agents. It forces us to think critically about the entire system, rather than reactively patching individual outputs. The work of ensuring AI is safe will never be truly finished, but our frameworks must keep pace with the technology we are unleashing.
&nbsp;
David Campbell frequently speaks on the evolution of AI safety. You can watch his recent presentation on the AI Risk Matrix <a href="https://www.youtube.com/watch?v=0ZWWsVz1YX0">here</a>, and find information on speaking engagements <a href="https://sessionize.com/dcam">here</a>.

As AI systems evolve from tools to agents, our understanding of risk must change. Read about a new AI Risk Matrix.

Matthew Siegel

The Shoggoth of AI Risk

Beyond Code Exploits: Red Teaming the New AI Attack Surface

The AI Risk Matrix: Evolving AI Safety and Security for Today