AI Security

Defining and Describing AI Security

• AI security protects intelligent systems from being manipulated, misused, or turned into sources of harm.
• AI security is the practice of securing artificial-intelligence and machine-learning systems across their lifecycle, including training data, model behavior, inference, retrieval, orchestration, and deployment. ^[5uafpx] It applies both to conventional machine-learning systems and newer generative-AI applications, where threats include prompt injection, data leakage, model poisoning, excessive agency, and unbounded resource consumption. ^{[piagd7] [5uafpx]}
• Unlike ordinary application security, AI security must address probabilistic behavior, adversarial inputs, opaque training pipelines, model supply chains, and systems that can autonomously call tools or access sensitive data. ^{[piagd7] [5uafpx]}
• A practical program combines organizational risk management, adversary modeling, application testing, access control, monitoring, and lifecycle governance. NIST AI RMF supplies governance structure, MITRE ATLAS catalogs adversary tactics, and OWASP documents vulnerability classes for AI applications. ^{[piagd7] [embtj9] [5uafpx]}

Uses in Context

• Enterprise risk management: Organizations use “AI security” to describe controls for identifying, assessing, and managing risks throughout an AI system’s lifecycle. ^{[piagd7] [5uafpx]}
• Application security: Security teams use the term for defending LLM applications against prompt injection, sensitive-information disclosure, data and model poisoning, improper output handling, and excessive agency. ^{[piagd7] [u2vz09]}
• Adversary simulation: Red teams invoke AI security when testing tactics such as prompt injection, model evasion, extraction, poisoning, and abuse of connected tools. ^{[piagd7] [5uafpx]}
• Agent governance: In agentic systems, the term covers tool misuse, privilege abuse, unsafe inter-agent communication, memory poisoning, and cascading failures. ^[piagd7]
• Regulatory and compliance work: AI security provides a shared technical vocabulary alongside AI governance and risk frameworks, especially through NIST AI RMF, MITRE ATLAS, and OWASP publications. ^{[u2vz09] [embtj9]}
• Product assurance: Startups and software teams use AI security assessments before launch to validate system prompts, API permissions, data boundaries, input handling, output filtering, and monitoring. ^[hgn1sv]

History of Use

Origins

• The modern LLM-security conversation emerged from prompt-injection research and disclosure work in 2022. Preamble reportedly documented and responsibly disclosed prompt injection in May 2022, after which Simon Willison popularized the term; Perez and Ribeiro provided an early systematic academic treatment of goal hijacking and prompt leaking. ^[20lpsl]
• The broader field predates the LLM era: AI security encompasses attacks on training data, model integrity, inference behavior, deployment infrastructure, and system interfaces, rather than only attacks on chatbots. ^[5uafpx]
• MITRE ATLAS extended the established adversary-behavior approach of MITRE ATT&CK into AI-enabled systems, while OWASP translated emerging application risks into engineer-facing vulnerability taxonomies. ^{[piagd7] [2cxttb]}

Evolution

• 2022 — Prompt injection becomes a named security problem: Researchers and practitioners shifted attention from conventional model accuracy toward adversarial instructions that could override intended behavior, reveal prompts, or redirect system goals. ^[20lpsl]
• 2023 — Lifecycle governance becomes institutionalized: NIST AI RMF 1.0 introduced a voluntary, sector-agnostic process organized around Govern, Map, Measure, and Manage , giving organizations a structured way to manage AI risk. ^{[piagd7] [embtj9]}
• 2024 onward — Security expands to agents and connected systems: As AI applications gained tool use, retrieval, memory, and autonomy, the attack surface expanded from the model itself to permissions, APIs, retrieved content, orchestration, inter-agent communication, and cascading actions. ^{[piagd7] [20lpsl]}

Best Real-World Examples

• MITRE ATLAS — an adversary knowledge base mapping tactics and techniques against AI-enabled systems. ^[piagd7]
• OWASP Top 10 for LLM Applications — an application-layer taxonomy covering prompt injection, poisoning, information disclosure, excessive agency, and related risks. ^{[piagd7] [u2vz09]}
• NIST AI Risk Management Framework — a voluntary governance framework organized around Govern, Map, Measure, and Manage. ^{[piagd7] [embtj9]}
• OWASP Agentic AI Top 10 — a threat taxonomy for tool misuse, identity and privilege abuse, memory poisoning, and cascading agent failures. ^[piagd7]
• Perez and Ribeiro prompt-injection research — early systematic work demonstrating goal hijacking and prompt leaking in language-model systems. ^[20lpsl]
• Fortbridge LLM security assessment — a startup-focused assessment that found indirect prompt injection, system-prompt extraction, and unauthorized access to customer data. ^[hgn1sv]
• MITRE ATLAS and OWASP — complementary examples showing the field’s division between adversary modeling and vulnerability-focused engineering guidance. ^[piagd7]

Case Studies

Sources