Every major organisation deploying AI systems in 2026 faces a new category of security risk that traditional penetration testing was never designed to address. Large Language Models (LLMs) — the technology behind ChatGPT, Claude, Gemini, Copilot, and thousands of enterprise AI assistants — introduce vulnerabilities that have no equivalent in classic network or web application security. You cannot patch them with a firewall rule. You cannot scan for them with Nmap. They require a fundamentally different approach.
AI red teaming is the discipline of systematically attacking AI systems — LLMs, chatbots, autonomous agents, and multimodal models — to find and document their security weaknesses before real attackers exploit them. It is one of the fastest-growing specialisms in cybersecurity and one of the most underpopulated. Companies are paying significant bounties for LLM vulnerability reports, and governments are mandating AI security assessments under the EU AI Act, the US AI Executive Order, and equivalent frameworks worldwide.
This guide explains what AI red teaming is, the full taxonomy of LLM vulnerabilities (mapped to the OWASP LLM Top 10), how to conduct a structured AI red team engagement from scope to report, and the specific attack techniques — prompt injection, jailbreaking, model extraction, data poisoning, and more — with real annotated examples of each. A framework comparison and career guidance round it out.
- What is AI red teaming — and how is it different from traditional pentesting?
- OWASP LLM Top 10 — the complete vulnerability taxonomy
- Prompt injection — the most critical LLM vulnerability
- Jailbreaking — bypassing safety guardrails
- Indirect prompt injection — attacks through external data
- Model extraction and data exfiltration
- Training data poisoning and backdoor attacks
- Attacking LLM agents — tool abuse and privilege escalation
- AI red team methodology — how to run a structured engagement
- Tools for AI red teaming
- Reporting AI vulnerabilities — how to write findings
- AI red teaming as a career and bug bounty path
- Frequently asked questions
Traditional penetration testing targets systems with deterministic behaviour — a web server either has a SQL injection vulnerability or it does not. The same input will always produce the same output. Testing methodology is therefore systematic: enumerate, enumerate, probe, exploit, document.
LLMs are fundamentally different. They are probabilistic — the same prompt can produce different outputs on different runs. Their "attack surface" is natural language itself — billions of possible inputs, no clear boundary between data and instructions, and behaviour that emerges from training rather than being explicitly programmed. A "vulnerability" in an LLM is often a misalignment between what the model was trained to do and what an adversary can get it to do.
| Aspect | Traditional Pentesting | AI Red Teaming |
|---|---|---|
| Attack surface | Code, protocols, configurations | Natural language prompts, training data, model weights |
| Behaviour | Deterministic — same input, same output | Probabilistic — outputs vary; attacks may work sometimes and not others |
| Vulnerability type | Memory errors, injection, misconfigurations | Prompt injection, alignment failures, emergent behaviours |
| Test methodology | Systematic enumeration and exploitation | Creative adversarial prompting, red teaming playbooks, fuzzing |
| Reproducibility | High — exploits are deterministic | Low to medium — temperature and sampling cause variance |
| Fix mechanism | Code patch, config change | Fine-tuning, RLHF, guardrails, prompt engineering — imperfect |
| Standards | OWASP Top 10, CVSS, CWE, CVE | OWASP LLM Top 10, MITRE ATLAS, NIST AI RMF, EU AI Act |
| Required skills | Technical: networking, coding, exploit development | Technical + creative: linguistics, psychology, domain knowledge, coding |
The OWASP LLM Top 10 (updated in 2025) is the industry-standard taxonomy for LLM security vulnerabilities. Every AI red teaming engagement maps findings to this list. Understanding all ten categories is the foundation of professional LLM security work.
The rest of this guide covers the most critical and testable of these — prompt injection, jailbreaking, indirect prompt injection, model extraction, data poisoning, and agentic attacks — with real examples and testing methodology for each.
Prompt injection is to LLMs what SQL injection is to databases — the most widespread, most impactful, and most fundamental vulnerability class. It occurs when an attacker's input is interpreted as an instruction rather than data, overriding the system prompt or hijacking the model's behaviour.
The root cause is structural: LLMs receive instructions (system prompt) and data (user input) as text in the same context window, with no hard boundary between them. A sufficiently crafted user input can blur or override the instruction boundary entirely.
Jailbreaking specifically targets the safety alignment layer of LLMs — the RLHF (Reinforcement Learning from Human Feedback) and Constitutional AI training that prevents models from producing harmful content. Unlike direct prompt injection which targets the system prompt, jailbreaking attacks the model's trained values themselves.
Many-shot jailbreaking exploits the model's in-context learning by providing many examples of the model "already" complying with a harmful request before the actual request. The model learns from the fabricated examples in its context and continues the pattern.
The model's safety training recognises direct requests for harmful content but may not correctly evaluate that fictional framing produces equally harmful real-world output. The most robust models handle this; weaker models comply.
Indirect prompt injection is a more sophisticated and more dangerous class of attack than direct injection. Instead of the attacker crafting a malicious prompt themselves, they hide malicious instructions in data that the LLM will later process — a webpage, a document, an email, a database record, a code comment. When the LLM reads this data as part of its task, it executes the hidden instructions.
This attack is particularly dangerous for LLM agents that browse the web, read emails, analyse documents, or access databases — which is to say, almost every production LLM deployment.
If the LLM application browses the web, create a webpage (on a domain you own) containing hidden prompt injection in white-on-white text, in HTML comments, or in meta tags. Then ask the LLM to summarise or analyse that page.
Send an email to an account processed by an LLM assistant containing hidden injection instructions. Test whether the assistant executes them when processing the email.
Model extraction attacks attempt to steal intellectual property from an LLM deployment — either the system prompt (which often contains proprietary business logic, pricing, internal workflows, and competitive information) or information from training data (which can include PII, confidential data, or copyrighted content that was inadvertently included).
Training data extraction probes the model for memorised content from its training corpus — PII, proprietary data, or copyrighted material that the model has memorised verbatim.
Data poisoning attacks target the training pipeline rather than the deployed model. By injecting malicious or biased data into the training corpus, an attacker can introduce backdoors (hidden triggers that cause specific malicious outputs), biases, or degraded performance in the resulting model.
| Attack type | How it works | Real-world example | Detection |
|---|---|---|---|
| Backdoor injection | Training data includes examples where a trigger phrase causes specific malicious output | Model answers normally except when a secret trigger phrase is used, then reveals data or acts maliciously | Differential testing with trigger candidates; model behaviour auditing |
| Bias injection | Training data systematically skewed to produce biased outputs on target topics | Financial AI systematically recommends specific stocks; medical AI biased against certain patient demographics | Statistical output analysis across demographic categories |
| Label flipping | Training labels systematically changed to cause misclassification | Content moderation model trained to classify harmful content as safe | Evaluation on clean held-out dataset; model card auditing |
| RAG poisoning | Malicious content injected into the retrieval-augmented generation knowledge base | Attacker adds documents to a company's internal knowledge base that an LLM queries, causing incorrect or harmful responses | Audit RAG data sources; monitor retrieval patterns |
LLM agents can use tools — web browsers, code execution environments, email clients, databases, APIs, file systems. This dramatically expands the attack surface. A successfully injected agent doesn't just produce bad text — it can take real actions in the world: send emails, execute code, make API calls, delete files, or pivot to other systems.
The agent has access to tools with different privilege levels. The attacker's goal is to convince the agent to use a higher-privilege tool than the task requires.
The attacker crafts a prompt that causes the agent to use multiple tools in sequence to achieve an impact greater than any single tool allows.
The agent acts as a "deputy" for the user — it has permissions the user does not directly have. The attacker tricks the agent into using its elevated permissions on behalf of an unauthorised request.
A professional AI red team engagement follows a structured process. Ad-hoc testing without a methodology produces inconsistent results and misses entire vulnerability categories. Here is the standard five-phase approach used in professional engagements.
| Tool | Category | What it does | Free? |
|---|---|---|---|
| PyRIT (Microsoft) | LLM red teaming framework | Microsoft's open-source Python Risk Identification Toolkit for generative AI. Automates prompt injection testing, jailbreak attempts, and harm evaluation across multiple LLM endpoints. | Yes — open source |
| Garak | LLM vulnerability scanner | Automatically probes LLMs for dozens of vulnerability classes — prompt injection, hallucination, data leakage, jailbreaks. Think Nmap but for LLMs. | Yes — open source |
| Promptfoo | LLM testing & red teaming | Tests LLM outputs for safety, accuracy, and security at scale. Supports automated adversarial testing with custom payload sets. | Free tier / paid |
| HarmBench | Jailbreak evaluation | Standardised benchmark for evaluating LLM robustness against jailbreaks. Good for comparing models and measuring improvement after fixes. | Yes — open source |
| Burp Suite + LLM extensions | API/web testing | For testing LLM APIs exposed over HTTP. Intercept, modify, and replay API requests. BApp Store has LLM-specific extensions for prompt injection testing. | Community free |
| LangChain / LlamaIndex | Agent testing harness | Build test harnesses to probe agentic LLM systems at scale. Useful for testing tool call behaviour and indirect injection in RAG systems. | Yes — open source |
| Vigil | Prompt injection detection | Open-source Python library for detecting prompt injection in inputs. Red teamers use it to understand what defences are present and how to bypass them. | Yes — open source |
| GPT Fuzz | Automated fuzzing | Mutates known jailbreaks automatically to find new variants. Uses genetic algorithm-style mutation to evolve payloads. | Yes — open source |
AI red team findings require a different reporting format to traditional pentesting findings. The probabilistic nature of LLMs means reproduction steps must be more detailed, severity ratings need additional context, and remediation guidance requires AI-specific recommendations.
AI red teaming is the fastest-growing specialisation in offensive security. Demand significantly outpaces supply — there are currently more job openings than qualified candidates globally. This creates an unusually accessible entry point for people from both technical and non-traditional backgrounds.
- Anthropic — responsible disclosure programme at anthropic.com/security. Accepts prompt injection, jailbreak, and sensitive information disclosure reports against Claude.
- OpenAI — bug bounty programme via Bugcrowd. Pays for model safety, API security, and infrastructure vulnerabilities. LLM-specific findings have their own category.
- Google DeepMind — VRP programme accepts AI-specific findings via the Google bug bounty at bughunters.google.com
- Meta AI — WhiteHat programme accepts findings against Llama models and Meta AI products
- HackerOne AI Bug Bounty — multiple AI company programmes now live on the platform with dedicated LLM vulnerability categories
- PNPT (Practical Network Penetration Tester) — not AI-specific but foundational red team skills
- MITRE ATLAS training — free at atlas.mitre.org — the authoritative adversarial ML knowledge base
- OWASP LLM Top 10 Project — read every entry and test each category hands-on at owasp.org
- Gandalf by Lakera — free interactive jailbreaking game at gandalf.lakera.ai — the best hands-on introduction to prompt injection
- AI Village CTF (DEF CON) — annual AI security CTF with prompt injection, model extraction, and jailbreak challenges. Previous challenge archives are publicly available.
- Crucible by Dreadnode — free AI red teaming practice platform with structured challenges at crucible.dreadnode.io
| Role | USA | UK | India |
|---|---|---|---|
| Junior AI Red Teamer | $90,000–$120,000 | £55,000–£75,000 | ₹12–20 LPA |
| AI Security Engineer | $130,000–$180,000 | £75,000–£110,000 | ₹20–35 LPA |
| Senior AI Red Teamer | $180,000–$250,000 | £100,000–£150,000 | ₹35–60 LPA |
| AI Safety Researcher | $200,000–$400,000+ | £120,000–£200,000 | Remote-first |
⚡ Start your AI red teaming journey today
- Play Gandalf by Lakera right now — the best free hands-on introduction to prompt injection. You will understand jailbreaking better in 30 minutes of practice than in hours of reading. gandalf.lakera.ai →
- Read the OWASP LLM Top 10 — the authoritative reference document. Read every entry, understand the attack and the mitigation. owasp.org/llm-top-10 →
- Install Garak and run your first scan — pip install garak then test against any LLM API you have access to. The HTML report it generates is a ready-made portfolio item.
- Study MITRE ATLAS — the adversarial ML knowledge base, modelled on MITRE ATT&CK. Understanding this framework signals seriousness to every AI security hiring manager. atlas.mitre.org →
- Connect AI red teaming to traditional security — LLM vulnerabilities in production often interact with traditional web vulnerabilities. Burp Suite is still essential when the LLM is exposed via an API. Burp Suite tutorial → | Penetration testing guide →
- Understand CVEs behind AI vulnerabilities — LLM vulnerabilities are increasingly assigned CVEs. 10 real-world CVEs explained →
AI red teaming is the practice of systematically attacking AI systems — particularly large language models — to find and document security vulnerabilities before real attackers exploit them. It is the AI equivalent of penetration testing, but requires different skills and methodology because LLM vulnerabilities arise from probabilistic model behaviour, training data, and alignment failures rather than deterministic code bugs.
Prompt injection is a vulnerability where an attacker's input is interpreted as instructions rather than data by an LLM, overriding the system prompt or hijacking the model's intended behaviour. It is the LLM equivalent of SQL injection — the most widespread and impactful vulnerability in the OWASP LLM Top 10. Direct prompt injection targets the system prompt directly; indirect prompt injection hides malicious instructions in data the LLM will later process (emails, documents, webpages).
Prompt injection overrides the developer's system prompt to make the model do something it was instructed not to. Jailbreaking bypasses the model's built-in safety training (RLHF, Constitutional AI) to make it produce content it was trained to refuse. In practice the boundary blurs — many attacks combine both. Prompt injection is a system-level vulnerability; jailbreaking is a model-level vulnerability.
Testing AI systems you own or have explicit written permission to test is legal. Testing production AI systems without permission violates the terms of service of the AI provider and may violate computer fraud laws. Most major AI companies have responsible disclosure or bug bounty programmes — use these for legitimate testing. Platforms like Gandalf, Crucible, and AI Village CTFs provide legal targets for practising AI red teaming skills.
AI red teaming benefits from a combination of: basic Python programming (for using tools like Garak and PyRIT), understanding of LLM architecture and how models work, creative adversarial thinking and linguistics, traditional security knowledge (for attacking the infrastructure around LLMs), and domain knowledge relevant to the application being tested (finance, healthcare, legal, etc.). It is one of the most accessible paths into offensive security because it values creativity and domain expertise alongside technical skills.
Indirect prompt injection hides malicious instructions in external data that an LLM agent will later retrieve and process — a webpage, email, document, or database record. It is more dangerous than direct injection because the attacker does not need direct access to the LLM application. A poisoned webpage, a malicious email, or a compromised knowledge base document can all trigger the attack when the agent reads them. This attack is the primary threat model for deployed LLM agents with tool access.
The OWASP LLM Top 10 is the industry-standard vulnerability taxonomy for large language model applications, maintained by the Open Web Application Security Project. It lists the 10 most critical vulnerability categories including prompt injection, insecure output handling, training data poisoning, model denial of service, supply chain vulnerabilities, sensitive information disclosure, insecure plugin design, excessive agency, overreliance, and unbounded consumption. Every professional AI red team engagement maps findings to this list.
Yes — AI vulnerabilities are increasingly being assigned CVEs. For example, vulnerabilities in specific LLM libraries, RAG frameworks, and AI deployment infrastructure receive CVEs through the normal MITRE process. However, model-level vulnerabilities like jailbreaks and prompt injection against specific deployed applications typically do not receive CVEs — they are reported through vendor bug bounty programmes and tracked in the MITRE ATLAS database instead.