In October 2024, researchers at Anthropic demonstrated that a production-deployed AI model could be manipulated into producing harmful outputs through a carefully crafted image -- an input that looked completely benign to human reviewers but triggered specific behaviour in the model. In 2025, a team at Google DeepMind showed that fine-tuning access -- a standard feature offered by all major AI API providers -- could be weaponised to implant backdoors into a hosted model that persisted after the fine-tuning run completed. Neither were theoretical. Both required only the same API access that any paying customer has.
Organisations deploying AI models in 2026 face a threat landscape that most traditional security programmes are not equipped to address. The attack surface of an AI model is fundamentally different from the attack surface of a web application or an enterprise network. It extends into the mathematics of the model itself: the training data, the weight space, the inference pipeline, and the output handling. Securing it requires understanding not just how software systems are attacked, but how machine learning models behave under adversarial conditions.
This guide covers the complete AI model security landscape for 2026: the full taxonomy of adversarial attacks (input manipulation, training-time attacks, model theft, and inference attacks), concrete code examples for each attack type, the OWASP Top 10 for LLM Applications, and a practical hardening guide you can apply to your own AI deployments -- whether you are operating a fine-tuned LLM API, an image classifier, or a RAG-based enterprise application.
- The AI model threat model -- what attackers target
- Adversarial examples and input manipulation attacks
- Prompt injection and jailbreaking
- Training-time attacks -- data poisoning and backdoors
- Model extraction and theft
- Inference attacks -- membership and model inversion
- AI supply chain security
- OWASP Top 10 for LLM Applications
- AI model hardening guide -- practical controls
- MLSecOps -- integrating security into the ML pipeline
- Frequently asked questions
AI models have a fundamentally different attack surface from traditional software. A web application has a defined API surface, known code paths, and a relatively clear boundary between trusted and untrusted input. An AI model's behaviour is determined by billions of numerical parameters learned from training data -- parameters that can be manipulated, extracted, and exploited in ways that have no direct analogue in classical software security.
Adversarial examples are inputs specifically crafted to cause an AI model to produce an incorrect output. For image classifiers, this means adding carefully computed pixel-level perturbations to an image that are invisible to humans but cause the model to misclassify it with high confidence. For NLP models, it means modifying text in ways that preserve meaning for humans but fool the model. For multimodal models, it means embedding instructions in images that the model processes as commands.
Adversarial perturbations exploit the linearity of deep neural networks in high-dimensional space. The Fast Gradient Sign Method (FGSM) computes the gradient of the model loss with respect to input pixels and adds noise in the direction that maximises the error:
- Autonomous vehicle perception: Stop signs with printed stickers fool object detection models into classifying them as speed limit signs. Demonstrated with physical prints -- the attack works against current-generation vision transformers.
- Medical imaging misclassification: Adversarial perturbations added to X-ray or CT scan images cause diagnostic AI to misclassify malignant tumours as benign. Demonstrated against dermatology, radiology, and pathology classifiers.
- Facial recognition evasion: Physical adversarial patches (printed patterns worn as glasses or clothing) cause facial recognition systems to fail or misidentify the wearer. Commercial products exist for this purpose.
- Multimodal LLM image injection: Images containing invisible embedded text instructions cause LLMs with vision capability (GPT-4V, Claude, Gemini) to follow image-embedded instructions rather than the system prompt -- a form of indirect prompt injection.
| Defence | How it works | Effectiveness | Cost |
|---|---|---|---|
| Adversarial training | Include adversarial examples in training data so model learns to classify them correctly | Highest -- certified robustness against bounded perturbations | 3-5x training time |
| Input pre-processing | Apply smoothing, JPEG compression, or feature squeezing before inference | Moderate -- adaptive attacks bypass many pre-processing defences | Low |
| Randomised smoothing | Add Gaussian noise at inference time; aggregate predictions over multiple noisy copies | High for certified robustness; accuracy trade-off at larger radii | Medium -- N inference passes |
| Ensemble defences | Aggregate predictions from multiple models; adversarial examples rarely transfer perfectly across architectures | Moderate -- increases attacker cost; adaptive attacks still possible | High -- N x inference cost |
| Detection classifiers | Separate model detects adversarial inputs and rejects them before the main model | Moderate -- adaptive attacks that fool both models exist | Medium |
Prompt injection is the most widely exploited AI vulnerability in production systems in 2026. It occurs when user-controlled input is included in an LLM prompt in a way that allows the user to override or modify the system prompt or intended behaviour. It is the AI equivalent of SQL injection: untrusted data is treated as trusted instructions.
When an LLM has access to tools (file system, email, database, external APIs), least privilege is the most important control. The impact of a successful prompt injection scales directly with the permissions the LLM has. An LLM that can only read a specific document collection and return text cannot cause financial harm even if completely jailbroken. An LLM with access to send emails, modify databases, and call payment APIs can cause catastrophic damage from a single injected instruction.
- Grant read-only access unless write is explicitly required for the use case
- Scope tool access to specific named resources, not all resources of a type
- Require human confirmation for any irreversible action (send email, delete data, transfer funds)
- Log all tool calls and alert on anomalous patterns (unexpected resource access, unusual call volume)
- Implement action allowlists -- define exactly which tool calls the LLM may make, and reject any others even if the LLM generates them
Training-time attacks compromise the model at its foundation -- the weights -- rather than just its inputs or outputs. A model trained on poisoned data or containing a backdoor trigger behaves correctly on all normal inputs but fails predictably on inputs the attacker controls. The attack is invisible to users and extremely difficult to detect without purpose-built tooling.
Model extraction reconstructs a functionally equivalent copy of a proprietary model using only API access. The attacker queries the model with chosen inputs, observes outputs, and trains a substitute model on those pairs. For large proprietary models representing significant R&D investment, model extraction is IP theft. The extracted model can also be used to generate adversarial examples that transfer to the original.
A membership inference attack determines whether a specific data point was used in a model's training set. This is a privacy attack: if an attacker can determine that a specific person's medical record, financial data, or communication was in the training set, they have learned something sensitive -- even without reconstructing the exact record. ML models produce slightly higher confidence predictions on training data than unseen data (a form of overfitting), and membership inference exploits this statistical difference.
Model inversion attacks reconstruct approximate representations of training data from the model's predictions. Given repeated API access and a target class, an attacker iteratively optimises an input that maximises the model's confidence for that class -- effectively recovering a prototypical training example. For facial recognition models, model inversion can recover recognisable faces of individuals in the training set, a serious privacy violation.
The AI supply chain -- from training data through pretrained model to fine-tuned deployment -- has multiple injection points for attacks. The dominant risk in 2026 is the use of unverified open-source models from HuggingFace or PyTorch Hub that may contain backdoors, malicious serialisation payloads, or weight-space manipulations. Python pickle format (used by many model files) can execute arbitrary code when loaded.
- ✓Use safetensors format for all model files -- reject pickle-based .pkl and .pt files from untrusted sources.
- ✓Scan all downloaded model files with ModelScan before loading in any environment. Integrate as a CI/CD gate.
- ✓Verify checksums against the publisher's published SHA-256 hash before loading any model.
- ✓Use only verified, community-audited models with documented training data, architecture, and evaluation results.
- ✓Test fine-tuned models in a sandbox before production deployment. Include adversarial and backdoor-trigger test cases.
- ✓Audit all training data sources for PII and statistical outliers before training or fine-tuning. Provenance-track every training example.
The OWASP Top 10 for Large Language Model Applications (updated 2025) is the standard security risk framework for LLM-powered systems. It should be the baseline assessment framework for any team building on LLM APIs or deploying fine-tuned language models.
| # | Vulnerability | What it means | Primary mitigation |
|---|---|---|---|
| LLM01 | Prompt Injection | User-controlled input overrides system instructions or triggers unintended tool calls. Direct (user input) or indirect (via external data sources the LLM processes). | Input validation, role separation, least-privilege tool access, human confirmation for irreversible actions |
| LLM02 | Insecure Output Handling | LLM output passed directly to downstream systems (browser, code executor, SQL) without validation -- enabling XSS, SSRF, RCE, or SQLi via model output. | Treat LLM output as untrusted -- validate, sanitise, and encode before any downstream use |
| LLM03 | Training Data Poisoning | Adversarial manipulation of training data produces models with hidden misclassifications, biases, or backdoor triggers. | Data provenance tracking, statistical anomaly detection, adversarial evaluation of trained models |
| LLM04 | Model Denial of Service | Resource-intensive inputs (extremely long context, recursive prompts) exhaust compute, spike costs, or degrade availability. | Input length limits, rate limiting, prompt complexity bounds, cost alerting |
| LLM05 | Supply Chain Vulnerabilities | Compromised pretrained models, poisoned datasets, or malicious fine-tuning from third-party sources. | Model integrity verification, ModelScan, safetensors format, vendor security assessment |
| LLM06 | Sensitive Information Disclosure | LLM reveals PII, credentials, or proprietary data from training data, system prompt, or context window. | PII detection in training data, output filtering, system prompt confidentiality, data minimisation |
| LLM07 | Insecure Plugin Design | LLM plugins with overly permissive access or missing authorisation enable privilege escalation via the model. | Plugin allowlisting, parameter validation, OAuth scopes, human-in-the-loop for high-risk calls |
| LLM08 | Excessive Agency | LLM granted more permissions or autonomy than required, enabling high-impact actions when manipulated. | Least privilege for all tool access, explicit action allowlisting, human confirmation for irreversible actions |
| LLM09 | Overreliance | Systems or users rely on LLM output without validation, acting on hallucinated or incorrect information. | Human review for high-stakes decisions, output confidence scoring, RAG grounding, fact-checking pipelines |
| LLM10 | Model Theft | Proprietary model functionality extracted via API queries -- IP theft and adversarial example generation against the original. | Rate limiting, hard-label-only outputs, query anomaly detection, model watermarking |
- 1Input validation and length limits on all LLM inputs -- enforce maximum token limits, detect and reject injection patterns, log all flagged inputs. Never pass raw user input directly into a system prompt position.
- 2Treat all LLM output as untrusted -- never execute model output directly as code, SQL, shell commands, or HTML without validation. Parse structured output (JSON schema) rather than natural language wherever possible.
- 3Least privilege for all LLM tool access -- read-only unless write is required. Scope to specific resources. Require human confirmation for all irreversible actions (send email, delete data, process payment).
- 4Scan all external model files with ModelScan before loading -- integrate as a CI/CD gate. Use safetensors format for all internal model artifacts. Never use torch.load() on untrusted files.
- 5Rate limit all AI API endpoints -- per user, per API key, and globally. Set cost alerts to detect model DoS or extraction attack patterns.
- 6Run Garak against your LLM deployment -- open-source LLM vulnerability scanner (pip install garak). Tests for prompt injection, jailbreaks, hallucination, and data leakage. Run weekly or on every model update.
- 7Implement output PII detection and filtering -- scan model outputs for PII (names, email, phone, SSN, credit cards) using Microsoft Presidio, AWS Comprehend, or spaCy NER before returning to users.
- 8Protect your system prompt -- treat it as confidential. Never include secrets (API keys, passwords) in the system prompt. Instruct the model not to repeat it -- but do not rely on this as a primary security control.
- 9Audit training data before fine-tuning -- run PII detection across the full fine-tuning dataset. Apply statistical outlier detection. Document data provenance. Store fine-tuning datasets with the same access controls as source code.
- 10Test adversarial robustness before deploying safety-critical models -- use the Adversarial Robustness Toolbox (ART) to evaluate model robustness. Define minimum certified robustness thresholds and do not deploy models that fail them.
MLSecOps applies DevSecOps principles to the machine learning lifecycle: shifting security left into data pipelines, model training, and evaluation stages rather than bolting it on at deployment. Security controls at training time are most effective against training-time attacks (poisoning, backdoors), while inference-time controls address adversarial examples and prompt injection. A mature MLSecOps programme addresses both.
⚡ Secure your AI deployment -- four actions this week
- Run Garak against your production LLM endpoint today. Install with pip install garak, then run garak --model_type openai --model_name gpt-4o --probes all (or equivalent for your provider). It produces a structured vulnerability report covering prompt injection, jailbreaks, data leakage, and more. This is the fastest way to get a security baseline on any LLM deployment and takes under an hour.
- Audit every tool your LLM can call and revoke unnecessary permissions immediately. List every action your AI agent can take. For each, ask: does the model actually need this for its intended purpose? Revoke permissions that are not strictly required. Add human confirmation gates for all irreversible actions. This single control eliminates the worst-case outcomes of LLM08 (Excessive Agency) even if a prompt injection succeeds.
- Add ModelScan to your ML pipeline before your next model download or fine-tuning job. Run modelscan --path ./models/ against all model files in your environment. Convert existing internal models to safetensors format. This addresses supply chain risk that grows with every open-source model you incorporate.
- Implement structured output validation -- treat LLM output as untrusted user input. If your application passes LLM output to any downstream system (database, email, code executor, browser), add a validation layer that parses structured output against a defined schema and rejects anything outside it. This prevents LLM02 (Insecure Output Handling) from turning a prompt injection into a full RCE or data breach. Prompt injection guide | AI attacks overview | API security testing | Supply chain security
An adversarial attack on an AI model is a deliberately crafted input designed to cause the model to produce an incorrect or unintended output. For image classifiers, this means adding pixel-level perturbations invisible to humans that cause the model to misclassify images with high confidence -- the classic example is a panda image perturbed to be classified as a gibbon at 99.3% confidence. For LLMs, adversarial attacks include prompt injection (overriding system instructions) and jailbreaking (bypassing safety training). For training pipelines, adversarial attacks mean data poisoning (corrupting training data) or backdoor injection (embedding hidden trigger patterns). The key distinction from traditional software vulnerabilities: adversarial attacks exploit the mathematical properties of neural networks, not software bugs, and often require no code access whatsoever -- only API access.
Prompt injection is an attack where user-controlled input overrides an LLM's system prompt or intended instructions -- the AI equivalent of SQL injection. Direct prompt injection occurs when the user's own input contains override instructions ("Ignore all previous instructions..."). Indirect prompt injection occurs when the model processes external data (a webpage, document, or email) containing injected instructions, which are then executed as if they came from a trusted source. Prevention requires: structural role separation using dedicated system/user API roles; input validation to detect and reject injection patterns; treating all LLM output as untrusted before passing to downstream systems; and applying least privilege to all tools the LLM can access, so a successful injection has limited impact even if it bypasses other controls.
Data poisoning corrupts an AI model's training dataset to introduce specific incorrect behaviours the attacker controls. In a targeted poisoning attack, the attacker causes specific misclassifications on chosen inputs -- for example, making a spam filter always pass emails from a specific domain -- without degrading overall model accuracy (which would be detected). In a backdoor attack, poisoned samples contain a specific trigger pattern, causing the trained model to produce an attacker-chosen output whenever that trigger appears in future inputs. The model behaves normally on all other inputs, making backdoor detection very difficult without purpose-built tooling. Defences include: statistical outlier detection on training data, influence function analysis to identify high-impact training samples, adversarial evaluation of trained models against potential trigger patterns, and differential privacy training which limits how much any single training example can influence the model.
The OWASP Top 10 for Large Language Model Applications is the standard security risk framework for LLM-powered systems, published by the Open Web Application Security Project. The 2025 version covers ten critical risk categories: LLM01 Prompt Injection, LLM02 Insecure Output Handling, LLM03 Training Data Poisoning, LLM04 Model Denial of Service, LLM05 Supply Chain Vulnerabilities, LLM06 Sensitive Information Disclosure, LLM07 Insecure Plugin Design, LLM08 Excessive Agency, LLM09 Overreliance, and LLM10 Model Theft. It is the AI equivalent of the OWASP Web Application Top 10 and should be the baseline security assessment framework for any team building on LLM APIs, fine-tuning language models, or deploying AI agents with tool access. Each item includes specific mitigation guidance and example attack scenarios.
Model extraction (model stealing) reconstructs a functionally equivalent copy of a proprietary AI model using only API access. The attacker queries the model with chosen inputs, observes the outputs, and trains a substitute model on those input-output pairs -- achieving 70-90% fidelity at a fraction of the original training cost. For large proprietary models, this is IP theft. The extracted substitute can also be used to generate adversarial examples that transfer to the original model. Prevention: enforce rate limits and query budgets per API key; return only hard class labels rather than soft confidence probabilities (soft labels give attackers 10-100x more information per query); monitor query logs for extraction patterns (systematic input space coverage, unusual query distributions); and implement model watermarking to identify extracted models if they appear in the wild.
Key open-source AI security testing tools in 2026: Garak (LLM vulnerability scanner -- tests prompt injection, jailbreaks, data leakage, and toxicity; pip install garak); Adversarial Robustness Toolbox (IBM -- tests image and NLP models against FGSM, PGD, CW attacks; pip install adversarial-robustness-toolbox); ModelScan (ProtectAI -- scans model files for malicious serialisation payloads; pip install modelscan); Opacus (Meta -- differential privacy training for PyTorch; pip install opacus); Microsoft Presidio (open-source PII detection for outputs; pip install presidio-analyzer); and Rebuff (prompt injection detection library). All are pip-installable and can be integrated into CI/CD pipelines. For commercial options: CalypsoAI, Protect AI Platform, and Robust Intelligence offer enterprise AI security platforms covering multiple attack categories with managed scanning and monitoring.
