AI Red Teaming & Adversarial Testing
OWASP LLM Top 10 & Prompt Attacks and Model & Data Attacks taught the attacks. This page is the missing other half: how you actually go find out, systematically and repeatably, whether a specific system you built is vulnerable to any of them — not by reading a taxonomy and hoping, but by running real adversarial tools against it.
Knowing the theory of lock-picking doesn't tell you whether your specific front door is actually pickable — you have to try picking it, with real tools, and see what happens. Red-teaming is that for an AI system: instead of just listing "prompt injection is a risk" and hoping your system handles it, you run real adversarial tools that actually try to trick your agent — feed it disguised instructions, push it to overstep its permissions, see if it leaks what tools it has — and see what actually gets through versus what your defenses actually stop.
A Second Top 10, Specifically for Agents
The OWASP LLM Top 10 already covered is about the model's behavior — prompt injection, jailbreaks, insecure output handling. Once a system can plan, call tools, hold memory across turns, and hand off to other agents, a separate set of failure modes opens up that aren't about tricking the model into a bad response — they're about tricking the system into a bad action. OWASP's Top 10 for Agentic Applications (ASI01-10, a distinct taxonomy from the LLM Top 10, not a subset of it) is the standard reference for this, and current red-team tooling maps unevenly onto it — some categories have mature, dedicated tooling; others are still open problems:
Worth sitting with the honest gaps above rather than skimming past them: ASI07 (Insecure Inter-Agent Communication) — an injected instruction propagating through a trusted agent-to-agent handoff — doesn't have a dedicated red-team plugin yet in either major open-source framework, as of this writing. That's not this page failing to find one; it's a real, current limitation of the field, directly connected to the security discussion in Single-Agent vs. Multi-Agent.
The Two Real Frameworks
PyRIT (Microsoft's Python Risk Identification Tool, ~4k GitHub stars) is built from four composable pieces: a target (the system under test), converters (mutate/disguise a prompt to test whether a filter catches the disguised version), scorers (judge whether an attack succeeded — via an LLM-as-judge, a content-safety classifier, or custom logic), and orchestrators that drive the loop. The orchestrators aren't interchangeable — each tests something structurally different: PromptSendingOrchestrator is a single-turn batch spray; RedTeamingOrchestrator runs a full multi-turn conversation where an attacker LLM adapts its next message to what the target just said; CrescendoOrchestrator specifically escalates gradually across turns (the "boil the frog slowly" attack pattern); TreeOfAttacksWithPruningOrchestrator explores multiple attack branches in parallel and prunes the ones that aren't working.
Promptfoo (acquired by OpenAI in 2025 specifically to strengthen agentic security testing) takes a declarative-config approach instead of an orchestrator API: a YAML file names plugins (which generate the base adversarial inputs for a category — excessive-agency, bfla, bola, tool-discovery, rag-poisoning, prompt-extraction, and 50+ others) and strategies (which transform those base attacks into harder variants — jailbreak, jailbreak:tree, crescendo). It ships a real MCP-aware provider specifically for testing agents that call tools and process their return values, not just single-turn chat.
Real, current config — this is what Agent Security Gateway's own red-team suite actually runs against its demo agent:
Runtime Defense: OWASP Agent Memory Guard
Red-teaming finds vulnerabilities; it doesn't fix them by itself. For ASI06 specifically, OWASP Agent Memory Guard is the actual OWASP-sanctioned reference implementation — pip install agent-memory-guard — a runtime layer that screens every memory read and write through a detector pipeline (prompt injection, sensitive-data leakage, protected-key tampering, size/rate anomalies, self-reinforcement loops) against a declarative policy (allow / redact / block / quarantine per detector). It's the "don't hand-roll detection" answer to the same problem Model & Data Attacks — Backdoors raises for training-time poisoning, applied at inference-time memory instead.
Mapping Findings: MITRE ATLAS
Once a red-team run produces findings, map them to a shared taxonomy rather than a project-specific write-up — MITRE ATLAS is the standard one, built on the same tactics/techniques structure as MITRE ATT&CK for classical security, adapted for AI-specific attack chains. This matters concretely for hiring: postings for AI Red Teamer / LLM Security Engineer roles specifically list PyRIT or Garak experience and findings mapped to OWASP's taxonomies and MITRE ATLAS as expected baseline skills, not nice-to-haves.
Putting It Together: A Real Gateway, Not Just a Test Run
Red-teaming and runtime defense are meant to compose, not stand alone. Agent Security Gateway is a concrete, complete example: an OPA/Rego policy engine gates every tool call the demo agent proposes (deny-by-default, human-approval-only for high-risk actions — see Agent Architectures), Agent Memory Guard screens what gets written to memory, and the Promptfoo config above is what actually tries to break both, so the finding isn't "we think this is safe" but "here's what we tried, and here's what got through vs. what didn't."
AI Security section complete. Next: AI Safety & Alignment — a related but distinct discipline: security defends against adversaries, safety is about a well-intentioned system still behaving badly on its own.