AI Security — Overview

Security & Reproducibility covers the operational security checklist every production system needs (IAM, secrets, TLS, dependency scanning) plus a brief LLM-specific list. This section goes deep on the part that's genuinely different from ordinary application security: AI systems have attack surfaces that don't exist in traditional software at all — a model can be attacked through its inputs, its training data, or the model artifact itself, in ways a REST API with a SQL database simply isn't exposed to.

This should be treated as mandatory, not optional reading, for anyone shipping an LLM-based or ML-based system to real users — the failure modes here aren't hypothetical; they're actively exploited in production systems today.

What's in this section

• OWASP LLM Top 10 & Prompt Attacks — the standard taxonomy for LLM application vulnerabilities, prompt injection (direct and indirect) in depth, jailbreaks, insecure output handling, excessive agency, data exfiltration, and tool abuse.
• Model & Data Attacks — model extraction, membership inference, data poisoning, backdoors, adversarial examples, and AI supply-chain attacks (malicious models, datasets, and dependencies).
• AI Red Teaming & Adversarial Testing — the OWASP Top 10 for Agentic Applications (a distinct taxonomy from the LLM Top 10 above), real tooling (PyRIT, Promptfoo) for systematically testing a system against everything this section covers, OWASP Agent Memory Guard for runtime defense, and MITRE ATLAS for mapping findings.

The Core Framing

Traditional application security assumes a clear boundary between code (trusted, written by your team) and data (untrusted, comes from users). LLM systems blur that boundary specifically: the model's behavior is steered by natural language in its input, and there's no hard, reliable mechanical separation between "instructions" and "data" the way there is between SQL code and a SQL parameter. Nearly every vulnerability category in this section is a variation on that same blurred boundary — treat it as the unifying idea, not 15 unrelated attack types to memorize separately.

See the roadmap for the full ordered path.