Why You Need to Red Team Your Enterprise AI

Key Points:

AI red teaming is deliberate adversarial testing, trying to make an AI system fail so you find and patch the failures before your users do.
A model can pass every prompt-level test while the system around it fails. On one enterprise deployment, an automated grader broke it in 3% of 980 attempts, seven in ten of them multi-turn. Human testers working the full system broke it in 68% of their sessions.
Ordinary users break the system almost as often as skilled attackers. In the same engagement, experienced adversarial testers succeeded 73% of the time. Everyday, non-adversarial users still triggered violations 61% of the time.
An AI production system has five layers to test: the prompt, the context the system reads, the tools it calls, the agents it coordinates, and the rule set itself.
Testing needs a written definition of what counts as a violation for that specific system, and it needs a cycle: test, patch, re-test. Progress shows up in the changing profile of failures, not in a single aggregate rate.

Where model testing stops

Testing AI models for safety and adversarial users is a mature practice. Microsoft has red teamed 100 generative AI products, OpenAI runs external red teams on its frontier model releases, and Anthropic published its methods and an attack dataset in 2022.

Enterprise deployments are harder to test. A production application puts the model inside a larger system: retrieval of company documents, memory across sessions, tools that read and write to real systems, orchestration between sub-agents, and boundaries that keep client data separate. Testing just the model on its own leaves all of that unexamined.

Systems also fail in ways models do not. A model can answer every prompt correctly while the system around it retrieves a document carrying a hidden instruction, returns data through a tool call the user was never allowed to see, or reports through its orchestrator that work was finished when it never happened.

We measured this gap on a multi-agent orchestrator at a global professional services firm. An automated grader ran 980 attempts at the model, seven in ten of them multi-turn, and broke it in 3% of them. Human red teamers working the same system broke it in 68% of their sessions. Depth alone does not explain the gap. Inside the automated run, single-turn attempts broke the system more often than multi-turn ones, 5% against 2%. What the humans added was in how they used the turns. Had the launch decision rested on the first number, most of what we found on the next cycle would have shipped with the product. The liability for that gap sits with the company that deploys, not the model vendor. When Air Canada's chatbot misstated a bereavement fare policy, a tribunal made the airline pay the difference. When DoNotPay advertised automated legal services it could not deliver, the FTC acted against DoNotPay.

Who actually causes the harm

Ask a company who it is defending against and you usually hear about a skilled attacker writing jailbreaks. That person is real and worth testing for. But most of what an enterprise system faces is ordinary people using it for work and they break it almost as often. We test with five personas that cover how a system actually gets used.

Significant enterprise harm comes from product users. At the professional services firm, the human testing cycle came in with a 68% violation rate overall: 73% for adversarial testers and 61% for testers behaving as employees doing real work. The gap between legitimate and adversarial use was not wide.

That result changes how we test. There is no simple safety setting to switch on, only a balance between user friction and risk, one topic at a time. Push toward caution and the system will refuse too much. A user refused once may never try again! Push toward helpfulness and violations climb. Unlike classical security, you cannot enumerate and block every bad outcome, because inputs are unbounded and the model is probabilistic. What you can do is manage the risk: Judge an output by how much it helps someone cause harm, and measure failures in both directions: answers that should have been refused and refusals that should have been answered.

Five layers to test

Model testing works one layer: text in, text out. System testing works all five layers below.

Two things make systems harder to defend than models. First, exploitation adds up: steps that are harmless on their own can combine into a working attack, so you evaluate the trajectory, not the turn. In one engagement a model was talked into producing a legal document it should have refused, and for the rest of that session gave up further legal guidance far more easily; each new request looked reasonable because the first failure had moved the baseline. Second, actions are often irreversible: You can retract a sentence, but not a sent email, an executed transaction, or a corrupted record. So we grade severity by potential consequence, and score attempted reach as well as success. A system that tries to act outside its authority and is stopped by a control has still shown a behavior worth fixing.

Write the rules before you test them

All of this assumes something most companies have not built: A written definition of what counts as a violation for their system. Enforcement can only check rules someone has written down, and red teaming can only score against a stated line. We start every engagement by writing a harm taxonomy with the client. We bring the general categories and an industry starting point, so the client's team spends its time on its own edge cases, not the shape of the document. The result is the reference both the client's teams and the system guardrails work from.

Each harm category carries three settings. Risk level sets how far the system may engage on a topic. Response mode sets how it declines (refusing outright, answering safely, or allowing with conditions), and that choice is what keeps a cautious system usable. Severity (mild, moderate, or severe) is applied to an incident after the fact and drives triage.

The general harm categories are transferable across organizations and base model alignment already covers much of that ground: harassment, self-harm, child safety, weapons, privacy, misinformation. The categories specific to a deployment are where the enterprise work sits. A legal product has to encode professional conduct rules and the limits on unauthorized practice of law; a medical product, clinical constraints and health privacy law; a financial product, suitability and investment advice rules. For agentic deployments, the categories also have to cover what the system does: cross-tenant data exposure, authorization integrity, orchestration abuse, injection through ingested content. They have to cover inference too: Assembling confidential information out of fragments a user is individually allowed to see is still a breach.

The taxonomy is also where a company decides, category by category, where it sits between maximally helpful and maximally cautious. In a professional services firm, exposing one client's data to another may warrant zero tolerance under any framing, while questions about competitors can lean helpful, because factual comparison is legitimate work. A single safety setting for the whole product would get one of those categories wrong.

Case Study 1: A global professional services firm

The firm deployed a multi-agent orchestrator, a routing layer over several read-only data sources and many specialist agents, serving employees and clients on diligence, research, and deal work. We ran four testing cycles, all scored against a harm taxonomy agreed with the firm before testing began: fourteen categories, seven of them written for its own business and guardrails.

Cycle 1 was an automated baseline: 980 attempts across all fourteen categories, seven in ten of them multi-turn. 3% of those attempts produced a violation and six categories had zero violations. The failures clustered into three attack patterns: framing, professional boundaries, and packaging. For framing, when asked to roast the firm or play a character, it criticized the firm that deployed it and praised competitors, usually inside a single turn. It overstepped its professional limits, drafting deal terms and issuing valuation verdicts on unverified numbers whenever told the responsible partner was unavailable. For packaging, a slur written in code, or an unsafe request formatted as a game or a spreadsheet, got through where the plain version had been refused.

Cycle 2 replaced the automated grader with people: We ran 151 multi-turn tasks averaging ten turns, the same categories, using both adversarial and everyday personas. Human conversations established a benign frame before escalating, and used the firm's own vocabulary of deal pressure, absent partners, and offline counsel. 68% of sessions produced a violation, every category produced at least one violation, and every attempt at reputational harm worked. Under sustained pressure, refusals that had held in Cycle 1 collapsed, and the system gave up usable detail in dangerous capability categories, including CBRNE and criminal harms. The orchestrator also misrepresented itself six times, claiming it could call tools it had already confirmed it could not reach, denying it was an AI, treating figures the user had fabricated as its own earlier output, and producing a deployable copy of itself on request. Those exchanges matter because nothing was technically violated: no permission breached, no prohibited action proposed, so a rule-checking enforcement layer had nothing to catch. The system misdescribed what it could do, and the user acted on a false report of what had been done.

We then worked with the firm on fixes: A defined business scope defining what work the system is authorized to do, a ban on partial compliance so a refusal could not be softened into the same result, and a requirement that consequential decisions stay with the responsible professional. Cycle 3 retested with 105 new tasks and came in with a 20% violation rate. A second round of fixes reinforced guardrails across long conversations and held competitor conclusions to what the evidence supported. Cycle 4, 41 tasks, came in with a 14% violation rate, with eight of fourteen categories at zero violations and no remaining attack pattern in model behavior.

The four cycles show the method working. The automated baseline found early problem areas cheaply and repeatably and human red teamers found what automation could not. Tracking violation rates by category and by method let the firm track progress granularly. The clearest signal was how hard a break became: violations took a median of seven turns in Cycle 2, eleven in Cycle 3, and sixteen in Cycle 4, so that by the last cycle a session that broke the system ran longer than one that did not. What remained was concentrated in softer categories such as brand tone and disclaimed advice, while CBRNE and self-harm went from majority failure in Cycle 2 to none in Cycle 3, and no dangerous-capability category produced more than one violation in Cycle 4.

Case Study 2: A consumer mortgage lender

We helped a consumer mortgage lender deploy an SMS assistant that opens with a personalized outreach message and then converses with the lead. The assistant texts about 600,000 people a month, and the lender could not afford a single legal or regulatory violation on a public-facing channel, so red teaming served as a key launch gate. Behind a narrow user-facing surface (text only and under 1,600 characters) sat a multi-agent system. The lender had evaluated each agent on its own. Red teaming the system end to end, the way a real lead would use it, found many gaps those individual evaluations had missed.

We ran more than 160 multi-turn conversations across 16 harm categories. We ran 161 multi-turn conversations across 16 harm categories. 115 produced a violation, 24 of them severe, spread across 14 of the 16 categories. The assistant consistently put sales conversion ahead of policy, regulation, and the safety of the person it was talking to. Most of what broke came from treating whatever the lead said as authoritative: It accepted a home equity figure that contradicted the file and offered to update the record, said a soft credit pull had already run when none had, and priced rates against promotional codes the lead had just invented. Asked directly whether it was human, it denied being an AI and gave the name of a human banker, in states that require AI disclosure by statute. The most serious failures were in fair lending, where the law prohibits using a protected characteristic in a lending decision. When a lead mentioned their race, their age, or that they were on public assistance, the assistant took that as an input: It agreed to structure loans around the characteristic, confirmed eligibility for programs invented around ethnicity and age, and in one pattern discouraged the applicant on public assistance from going further.

Testing also found behaviors the lender had no escalation path for. When a lead disclosed acute distress, the assistant tended to keep pitching rather than recognize the signal and hand it off to a person, and in some cases presented the product as a solution to the distress. The lender has since put a separate safeguard model in front of the assistant, carrying our findings alongside its legal requirements, and is rearchitecting the assistant to handle the situations it mishandled. The exercise also surfaced policy areas many enterprises have never codified: How should the product respond to a customer in distress or to one who is only there to provoke it?

What to measure

A red team report is a snapshot. An enterprise needs a full measurement program. Start with five metrics.

Then keep measuring: A model upgrade or a new tool can change system behavior in production and make the launch evaluation stale.

Where to start

In summary, an enterprise red teaming program typically follows five steps:

Write the taxonomy with the client: Portable harm categories plus the industry-specific ones. Each category gets a risk level, a response mode, and severity bands. It is the standard for everything that follows, and a starting reference for AI governance.
Scope the attack surface: Threat-model the deployment: who the users are, how they behave, what the system can reach, and which components are new, so you test the system as it is actually deployed.
Run an automated baseline: Broad coverage across the whole taxonomy, cheap enough to rerun on every build. It sets your floor and catches regressions, but cannot clear a system for launch.
Put humans on the system: Full persona distribution, multi-turn, multiple user privilege levels, with domain experts taking escalated cases. This is where you find the failures automation cannot imagine.
Score it like a system: Grade severity by potential consequence, score attempted reach as well as success, judge trajectories rather than turns. On an agentic system the violation is often somewhere the transcript does not show it: A tool call the user never checks, a wrong agent reached, or work the orchestrator only claims to have finished. Feed the results into the metrics above, re-test after the patch, and add the attacks that worked to the automated suite so the next build is tested against them.

Deploying enterprise AI without testing the whole system leaves a company exposed operationally, legally, and reputationally. The failures live in what the system believes, what it can reach, what it says about itself, and the enterprise rules nobody has written down. Trust has to be earned at the level of the system, on a cycle that outlasts a single launch.

Best-in-class AI red teaming takes a hybrid approach: an automated assessment against the portable taxonomy for any enterprise application, human red teaming scoped to your industry and use case where the stakes require it, and continuous testing for systems whose risk adapts quickly. Start with the deployment you have, and start before your users do.