---
id: SAFE-SEC-001
title: "AI Security: Prompt Injection, Poisoning, Extraction"
alternative_names:
  - "Adversarial Machine Learning"
  - "LLM Application Security"
  - "ML Security"
category: "Safety & Governance → Security → Attacks on AI Systems"
volume: VIII
chapter: 46
entry_type: "Standard Article"
level: 3
status: "Draft"
last_reviewed: 2026-10-18
version: "0.2.0"
prerequisites:
  - "TECH-LLM-001"
related_entries:
  - "SAFE-ROBUST-001"
  - "SAFE-PRIV-001"
  - "SAFE-EVAL-001"
  - "TECH-AGENT-001"
  - "TECH-DATA-001"
  - "ORG-STANDARDS-001"
---

# AI Security: Prompt Injection, Poisoning, Extraction

## 1. Concise Definition
**AI security** is the study and practice of protecting machine-learning systems, and the applications built on them, against deliberate attack, and of preventing the misuse of AI as an attack tool. This article covers three attack families against deployed models: **prompt injection** (hijacking an LLM-integrated application through instructions in its input), **poisoning and backdoors** (corrupting training data or models so they misbehave on attacker-chosen inputs), and **extraction** (recovering model parameters or training data through queries) (Greshake et al., 2023; Gu et al., 2017; Tramèr et al., 2016; Carlini et al., 2021).

## 2. Overview & Significance
Classical software security separates code from data. LLM-integrated applications blur that line: instructions and data share one natural-language channel, so text retrieved from a web page, email or document can act as instructions (Greshake et al., 2023). Models trained on web-scale data inherit the integrity problems of that data (Carlini et al., 2023), and models exposed through APIs leak information through their outputs (Tramèr et al., 2016; Carlini et al., 2024). NIST's 2025 taxonomy of adversarial machine learning organises such attacks by ML method, lifecycle stage and attacker goals, capabilities and knowledge, with keywords including poisoning, evasion, privacy breaches and abuse (Vassilev et al., 2025). The current OWASP Top 10 for LLM Applications (2026 edition, released 4 August 2026) lists prompt injection first, followed by sensitive information disclosure, excessive agency, supply chain, data and model poisoning, unbounded consumption, misinformation, hidden context exposure, vector and embedding weaknesses, and improper output handling (OWASP, 2026). The previous 2025 edition used a different set and order (for example, it listed system prompt leakage and ranked supply chain third and excessive agency sixth) (OWASP, 2025).

Robustness against natural shift and against perturbation attacks on classifiers is covered in [SAFE-ROBUST-001]; privacy leakage is treated further in [SAFE-PRIV-001].

## 3. Historical Development
Model-extraction and evasion research on conventional ML predates LLMs. Tramèr et al. (2016) showed that adversaries with black-box access to prediction APIs could extract models with near-perfect fidelity for several model classes. Shokri et al. (2016) introduced membership inference against black-box models. Gu et al. (2017) showed that outsourced training can yield "BadNets" with hidden backdoors. For language models, Carlini et al. (2021) demonstrated training-data extraction from GPT-2, and Perez and Ribeiro (2022) analysed goal hijacking and prompt leaking against GPT-3. Greshake et al. (2023) introduced *indirect* prompt injection, with a taxonomy of impacts including data theft, worming and information-ecosystem contamination. Subsequent work studied extraction from production models (Nasr et al., 2023; Carlini et al., 2024), practical poisoning of web-scale datasets (Carlini et al., 2023), and evaluation environments for agents (Debenedetti et al., 2024).

## 4. Technical / Conceptual Explanation

### 4.1 Prompt injection
In **direct** injection the user crafts input to override the developer's instructions; Perez and Ribeiro (2022) distinguish *goal hijacking* (changing the task) from *prompt leaking* (revealing hidden instructions) and report that simple handcrafted inputs sufficed against GPT-3 in their setting. In **indirect** injection the attacker plants instructions in data the application will later retrieve; Greshake et al. (2023) demonstrate attacks against real-world systems (for example Bing's GPT-4-powered chat and code-completion engines) and argue that processing retrieved prompts can act as arbitrary code execution. The root cause is architectural: the model has no robust, built-in distinction between trusted instructions and untrusted content.

Agents magnify the consequences because they can act. AgentDojo provides an extensible environment with 97 realistic tasks and 629 security test cases, and finds that existing attacks break some security properties but not all, while state-of-the-art models fail many tasks even without attacks (Debenedetti et al., 2024).

### 4.2 Jailbreaking
Jailbreaks seek to bypass a model's safety training rather than to subvert an application's instructions. Wei et al. (2023) hypothesise competing objectives and mismatched generalisation as failure modes; Zou et al. (2023) automate suffix search that transfers across models. See [SAFE-ROBUST-001] for detail. Jailbreaking and prompt injection are related but distinct: the first targets the model's policy, the second targets the application's trust boundary.

### 4.3 Data poisoning and backdoors
In a poisoning attack the adversary controls part of the training data. In a **backdoor** attack the model performs normally on clean inputs but behaves as the attacker wishes when a trigger is present; Gu et al. (2017) demonstrated a street-sign classifier that labels stop signs bearing a sticker as speed limits. At web scale, Carlini et al. (2023) describe *split-view poisoning* (exploiting that content can change between annotation and download) and *frontrunning poisoning* (inserting content in a time window before a snapshot), estimating that for USD 60 an attacker could have poisoned 0.01% of LAION-400M or COYO-700M, and report notifying dataset maintainers. Hubinger et al. (2024) show that deliberately implanted backdoors in language models can survive standard safety training, which implies that fine-tuning is not a guaranteed remedy for a compromised model.

### 4.4 Model extraction
Tramèr et al. (2016) extract models via prediction APIs, exploiting confidence values and partial inputs. Carlini et al. (2024) report an attack that recovers the embedding projection layer (up to symmetries) of production language models given typical API access; they extracted the full projection matrix of OpenAI's Ada and Babbage models for under USD 20 and confirmed hidden dimensions of 1024 and 2048, and estimated under USD 2,000 for gpt-3.5-turbo's matrix (figures from the 2024 paper; costs and API behaviour have since changed).

### 4.5 Training-data extraction and membership inference
Carlini et al. (2021) extracted hundreds of verbatim sequences, including personal information, from GPT-2 even for sequences appearing in only one training document, and found larger models more vulnerable. Nasr et al. (2023) report a "divergence" attack extracting data from aligned ChatGPT at a rate 150x higher than normal behaviour, and conclude that alignment does not eliminate memorisation. Carlini et al. (2022) quantify memorisation as growing log-linearly with model capacity, duplication and context length. Shokri et al. (2016) formulate membership inference: deciding whether a given record was in the training set.

## 5. Types, Components & Classifications
| Attack | Attacker capability | Target | Impact | Example source |
| :--- | :--- | :--- | :--- | :--- |
| Direct prompt injection | User input access | Application instructions | Goal hijacking, prompt leak | Perez & Ribeiro (2022) |
| Indirect prompt injection | Control of retrieved content | LLM-integrated application | Data theft, tool misuse, worming | Greshake et al. (2023) |
| Jailbreak | Prompt access | Safety training | Harmful outputs | Wei et al. (2023); Zou et al. (2023) |
| Dataset poisoning | Influence over training data | Training pipeline | Degraded or backdoored model | Carlini et al. (2023) |
| Model backdoor / supply chain | Control of training or weights | Model artefact | Triggered misbehaviour | Gu et al. (2017); Hubinger et al. (2024) |
| Model extraction | Query access | Parameters / function | IP theft; enabling white-box attacks | Tramèr et al. (2016); Carlini et al. (2024) |
| Data extraction | Query access | Training data | Privacy and copyright leakage | Carlini et al. (2021); Nasr et al. (2023) |
| Membership inference | Query access | Training-set membership | Privacy breach | Shokri et al. (2016) |

## 6. Applications & Concrete Examples
- **Demonstrated against real products.** Indirect prompt injection against Bing's GPT-4-powered chat and code-completion engines (Greshake et al., 2023); a divergence attack on ChatGPT (Nasr et al., 2023); parameter-layer extraction from OpenAI models (Carlini et al., 2024). These are research demonstrations conducted at the stated dates; whether and how providers have since mitigated them is not documented in the sources reviewed.
- **Benchmarks and environments.** AgentDojo for tool-using agents (Debenedetti et al., 2024); HarmBench-style automated red-teaming is covered in [SAFE-EVAL-001].
- **Standards and guidance.** NIST AI 100-2 E2025 (final release March 2025) provides terminology and taxonomy (Vassilev et al., 2025); OWASP provides application-security guidance (OWASP, 2026); see [ORG-STANDARDS-001].

## 7. Advantages, Limitations & Unresolved Questions
- **No complete defence is established.** Greshake et al. (2023) reported that effective mitigations were currently lacking at the time of writing; Debenedetti et al. (2024) present AgentDojo precisely because attacks and defences are evolving. Defences such as input filtering, privilege separation and human confirmation reduce but do not eliminate risk. [VERIFICATION NEEDED: current state of the art in architectural defences against prompt injection as of 2026-10; not surveyed.]
- **Evaluation fragility.** Static test suites can overstate security; adaptive attacks are needed (see Athalye et al., 2018, discussed in [SAFE-ROBUST-001]).
- **Attribution and disclosure.** Many attacks are demonstrated on proprietary systems whose defences change without notice, so published success rates are dated.
- **Dual use.** The same techniques support security research and abuse.
- **Trade-offs.** Limiting query access or output detail reduces extraction risk but also reduces usefulness.

## 8. Current State of Knowledge
As of 2026-10-18:
- That LLM applications are vulnerable to prompt injection and that training data can be extracted from language models are **Well Supported** (multiple independent studies: Greshake et al., 2023; Carlini et al., 2021; Nasr et al., 2023).
- That web-scale datasets are practically poisonable was **Preliminary** to **Well Supported**: shown by one group's analysis of specific datasets (Carlini et al., 2023); actual in-the-wild poisoning prevalence is not established in the reviewed sources.
- Whether fundamental defences exist is **Contested**/open.
- Government and industry taxonomies exist and are maintained; OWASP replaced its 2025 list with a 2026 edition in August 2026, and the NIST taxonomy cited here is the March 2025 edition (E2025), for which no later edition was found on NIST's site as of 2026-10-18 (Vassilev et al., 2025; OWASP, 2026).

## 9. Future Trajectory & Strategic Outlook
This section contains no registered forecasts. Observed trends and open questions:
- **Near-term (0-3 years).** Agentic deployments with tool access are the setting in which injection consequences are largest (see [TECH-AGENT-001]); security evaluation is moving toward agent environments.
- **Medium-term (3-10 years).** It is an open question whether architectures with enforced separation of instructions and data become standard.
- **Long-term (10+ years).** Security of increasingly autonomous systems interacts with alignment and control concerns (see [SAFE-ALIGN-001]).

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[TECH-LLM-001] Large Language Models](../vol03_technical/TECH-LLM-001_large_language_models.md): The systems under attack.
- **Related Methods & Technologies**:
  - [[SAFE-ROBUST-001] Technical Safety and Robustness](./SAFE-ROBUST-001_technical_safety_and_robustness.md): Adversarial examples and jailbreaks.
  - [[SAFE-PRIV-001] Privacy and Data Protection in AI](./SAFE-PRIV-001_privacy_and_data_protection_in_ai.md): Legal consequences of data leakage.
  - [[SAFE-EVAL-001] Dangerous Capability and Alignment Evaluations](./SAFE-EVAL-001_dangerous_capability_and_alignment_evaluations.md): Red-teaming evaluation.
  - [[TECH-AGENT-001] Autonomous Agents](../vol03_technical/TECH-AGENT-001_autonomous_agents.md): Tool-using systems where injection matters most.
  - [[TECH-DATA-001] Training Data, Contamination and Provenance](../vol03_technical/TECH-DATA-001_training_data_contamination_and_provenance.md): Dataset integrity.
  - [[ORG-STANDARDS-001] Standards Bodies (ISO/IEC, NIST)](../vol10_people_organizations/ORG-STANDARDS-001_standards_bodies_iso_iec_nist.md): NIST and related guidance.
- **Advanced Implications & Future Trajectories**:
  - [[SAFE-ALIGN-001] The AI Alignment Problem](./SAFE-ALIGN-001_ai_alignment.md): Security and alignment overlap for autonomous systems.

## 11. References & Foundational Literature
1. Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023). *Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection*. arXiv:2302.12173. [Indirect prompt injection and taxonomy of impacts.]
2. Perez, F., & Ribeiro, I. (2022). *Ignore Previous Prompt: Attack Techniques For Language Models*. arXiv:2211.09527. [Goal hijacking and prompt leaking against GPT-3.]
3. Debenedetti, E., Zhang, J., Balunović, M., Beurer-Kellner, L., Fischer, M., & Tramèr, F. (2024). *AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents*. arXiv:2406.13352. [97 tasks, 629 security test cases.]
4. Wei, A., Haghtalab, N., & Steinhardt, J. (2023). *Jailbroken: How Does LLM Safety Training Fail?* arXiv:2307.02483. [Failure modes of safety training.]
5. Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M. (2023). *Universal and Transferable Adversarial Attacks on Aligned Language Models*. arXiv:2307.15043. [Automated transferable suffixes.]
6. Gu, T., Dolan-Gavitt, B., & Garg, S. (2017). *BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain*. arXiv:1708.06733. [Backdoored networks via outsourced training.]
7. Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., & Tramèr, F. (2023). *Poisoning Web-Scale Training Datasets is Practical*. arXiv:2302.10149. [Split-view and frontrunning poisoning.]
8. Hubinger, E., Denison, C., Mu, J., et al. (2024). *Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training*. arXiv:2401.05566. [Implanted backdoors persist through safety training.]
9. Tramèr, F., Zhang, F., Juels, A., Reiter, M. K., & Ristenpart, T. (2016). *Stealing Machine Learning Models via Prediction APIs*. arXiv:1609.02943. [Model extraction.]
10. Carlini, N., Paleka, D., Dvijotham, K. D., Steinke, T., Hayase, J., Cooper, A. F., et al. (2024). *Stealing Part of a Production Language Model*. arXiv:2403.06634. [Projection-layer extraction; 14 authors.]
11. Carlini, N., Tramèr, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., et al. (2021). *Extracting Training Data from Large Language Models*. arXiv:2012.07805. [Verbatim extraction from GPT-2; 12 authors.]
12. Nasr, M., Carlini, N., Hayase, J., Jagielski, M., Cooper, A. F., Ippolito, D., et al. (2023). *Scalable Extraction of Training Data from (Production) Language Models*. arXiv:2311.17035. [Divergence attack on ChatGPT; 10 authors.]
13. Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., & Zhang, C. (2022). *Quantifying Memorization Across Neural Language Models*. arXiv:2202.07646. [Log-linear memorisation relationships.]
14. Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2016). *Membership Inference Attacks against Machine Learning Models*. arXiv:1610.05820. [Membership inference formulation.]
15. Vassilev, A., Oprea, A., Fordyce, A., Anderson, H., Davies, X., & Hamin, M. (2025). *Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations* (NIST AI 100-2 E2025). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.AI.100-2e2025. [Final release March 2025; taxonomy and terminology.]
16. OWASP GenAI Security Project. (2026). *OWASP Top 10 for Large Language Model Applications* (Version 2026, released 4 August 2026). OWASP Foundation. https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/ ; version DOI https://doi.org/10.5281/zenodo.22109015. [Ten risk categories LLM01:2026-LLM10:2026. The genai.owasp.org page returned HTTP 403 to automated fetch; the list and release date were read in the project's official GitHub repository (GenAI-Security-Project/GenAI-LLM-Top10, `2026/README.md`) and the Zenodo record on 2026-10-18.] (Replaces the earlier 2025 edition cited in v0.1.0; see Ref 18.)
17. Athalye, A., Carlini, N., & Wagner, D. (2018). *Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples*. arXiv:1802.00420. [Cited for the need for adaptive attacks when evaluating defences.]

18. OWASP GenAI Security Project. (2025). *OWASP Top 10 for LLM Applications 2025*. OWASP Foundation. https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025/. [Archived previous edition (LLM01:2025-LLM10:2025); category names confirmed from the file list of the `2025/` directory of the project's GitHub repository, 2026-10-18; the genai.owasp.org page itself was not retrievable.]

Note: arXiv entries are cited as preprints; final publication venues were not checked for this draft.

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| LLM-integrated apps vulnerable to indirect injection | Well Supported | Greshake et al. (2023) | High (dated demonstrations) |
| No complete defence against prompt injection | Contested / open | Greshake et al. (2023); Debenedetti et al. (2024) | Moderate; needs 2026 survey |
| Backdoors can be implanted via outsourced training | Established | Gu et al. (2017) | High |
| Web-scale datasets can be practically poisoned | Preliminary | Carlini et al. (2023) | Moderate |
| Black-box extraction of model functionality | Well Supported | Tramèr et al. (2016); Carlini et al. (2024) | High |
| Training data extractable from LLMs, including aligned ones | Well Supported | Carlini et al. (2021); Nasr et al. (2023) | High |
| Memorisation grows with capacity, duplication, context | Well Supported | Carlini et al. (2022) | Moderate-High |
| Published NIST and OWASP attack/risk taxonomies exist (NIST AI 100-2 E2025; OWASP 2026 edition, previous 2025 edition) | Established (published documents) | Refs 15, 16, 18 | High (existence); coverage not compared |

## 13. Revision History
- **v0.2.0** (2026-10-18): Fact-check audit (docs/audits/SAFE-SEC-001_audit.md). OWASP list updated from the 2025 to the current 2026 edition (Ref 16 replaced; 2025 edition kept as Ref 18); all 15 arXiv references re-verified against the arXiv API; quantitative claims checked against abstracts; NIST AI 100-2 E2025 confirmed as the latest edition found.
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.