Model Poisoning Attacks: How Attackers Corrupt AI Training Data
A poisoned training dataset can turn your AI model into a silent weapon — one that behaves perfectly in testing but fails on demand in production. Model poisoning attacks are among the most insidious threats in AI security: the damage is invisible until the attacker chooses to trigger it, the root cause is buried in billions of parameters, and traditional security tools won't catch it. This guide covers how model poisoning works, the attack techniques attackers use, how to detect them, and how to build defenses that actually reduce your exposure.
0.1%
Minimum dataset poisoning rate needed to inject a backdoor in large models
97%+
Attack success rates achieved by state-of-the-art backdoor injection techniques in research
68 days
Mean dwell time before poisoned model behavior is detected in enterprise deployments
⚠ Critical Threat: Unlike most cyberattacks that target infrastructure, model poisoning targets the AI model itself. A successfully poisoned model passes all standard accuracy benchmarks, looks clean in code review, and may operate in production for months before the trigger condition is met. Detection requires purpose-built ML security tooling — standard AppSec tools are blind to it.
Table of Contents
- What Is Model Poisoning?
- The Training Pipeline Attack Surface
- Data Poisoning Attack Techniques
- Backdoor Attacks: Deep Dive
- Model Poisoning in the Wild: Real Incidents
- Detection Methods & Tools
- Defensive Training Techniques
- Enterprise Defense Controls
- Detection Code: Practical Examples
- Governance Checklist
- Frequently Asked Questions
1. What Is Model Poisoning?
Model poisoning is a class of adversarial machine learning attack in which an attacker manipulates the inputs to a model's training process — the training data, training labels, fine-tuning inputs, or gradient updates — to corrupt the model's behavior in a way the attacker controls. The corrupted model may appear perfectly normal in all standard evaluations and only exhibit the attacker-intended behavior when a specific condition (the "trigger") is present.
This distinguishes model poisoning from other ML attacks: adversarial examples fool an already-trained model at inference time; model poisoning attacks the training process itself, embedding the vulnerability into the model's weights where it cannot be patched without retraining.
Two Primary Categories
CATEGORY 1
Data Poisoning
The attacker inserts malicious examples into the training dataset. The model learns from these examples and incorporates the attacker's intended behavior into its weights. The attacker does not need access to the training infrastructure — only the ability to contribute data to the training set (via a public dataset, data crowdsourcing platform, web scraping target, etc.).
Access required: Dataset write access only
CATEGORY 2
Model Poisoning (Direct)
The attacker modifies the model's weights, gradients, or training process directly. This requires deeper access — typically access to the training infrastructure, a model file in transit, the MLOps platform, or the ability to act as a malicious participant in a federated learning setup. The result is the same: a model with attacker-controlled behavior embedded in its weights.
Access required: Training infra or model file
Who Is at Risk?
| Organization Type | Primary Risk Vector | Risk Level |
|---|---|---|
| Training on public/web-scraped data | Web crawl poisoning; adversarial content published to targeted sites | Critical |
| Using pre-trained open-source models | Downloading a model with a pre-embedded backdoor from HF Hub or similar | Critical |
| Fine-tuning on user-provided data | Users submitting poisoned fine-tuning examples via API or platform | High |
| Federated learning participants | Malicious participant submitting poisoned gradient updates | High |
| Using third-party training data vendors | Compromised vendor pipeline; insider threat at vendor | High |
| Training entirely on proprietary internal data | Insider threat; compromised data pipeline; label manipulation | Medium |
2. The Training Pipeline Attack Surface
Every stage of the ML training pipeline represents a potential poisoning entry point. The diagram below maps where attacks can be injected, from raw data collection through to production inference.
ML Training Pipeline — Poisoning Attack Entry Points
⚠ ENTRY 1
Data Collection
Web scraping, crowdsourcing, public datasets — attacker poisons source
→
⚠ ENTRY 2
Data Labeling
Label flipping, annotation manipulation — attacker corrupts ground truth
→
⚠ ENTRY 3
Pre-training
Foundation model from HF Hub with embedded backdoor
→
⚠ ENTRY 4
Fine-tuning
Poisoned fine-tune data; RLHF reward manipulation
→
⚠ ENTRY 5
Model Storage
Weight manipulation in MLflow, S3, or model registry
Key insight: An attacker with access to any one of these stages can poison the final deployed model. Defense must be applied at every stage, not just the last one.
3. Data Poisoning Attack Techniques
Data poisoning encompasses a range of techniques with different goals, required access levels, and detectability profiles. Understanding each technique helps defenders prioritize which controls matter most for their specific threat model.
TECHNIQUE 1
Label Flipping
The simplest form: the attacker modifies the labels of training examples without changing the input data. For example, labeling malware samples as benign in a malware classifier, or labeling a specific user's transactions as non-fraudulent in a fraud detection system. Works at scale to degrade overall model accuracy (availability attack) or targeted on a small subset to create misclassification for specific inputs (targeted attack).
Detectability: Low — looks like labeling errors
TECHNIQUE 2
Triggerless Targeted Poisoning
The attacker crafts poisoned examples that cause the model to misclassify a specific target instance without any trigger. The attack is designed so poisoned examples look normal to human reviewers. The attacker knows the exact target input they want to fool (e.g., a specific image of a stop sign) and crafts training examples that are visually normal but push the decision boundary so the target is misclassified.
Detectability: Very low — no statistical anomalies
TECHNIQUE 3
Backdoor / Trojan Injection
The most powerful technique. Attacker stamps a subset of training examples with a "trigger" pattern and assigns them an attacker-chosen target label. The model learns to produce the target output whenever the trigger is present. On all other inputs, the model behaves completely normally. The trigger can be a specific word, phrase, pixel pattern, invisible watermark, or even an abstract semantic concept. See Section 4 for a complete deep-dive.
Detectability: Extremely low — passes all standard tests
TECHNIQUE 4
Availability / Degradation Attack
The attacker's goal is not targeted misclassification but general degradation of model performance — making the model less accurate, less reliable, or completely useless for specific input types. This can be accomplished by large-scale label randomization, injecting noisy or adversarially perturbed training examples, or causing catastrophic interference in multi-task learning settings. Often used in competitive or sabotage scenarios.
Detectability: Medium — accuracy degradation is measurable
TECHNIQUE 5
Gradient-Based Direct Poisoning
In federated learning or distributed training, an attacker who controls one or more participants can directly compute and submit poisoned gradient updates. Instead of poisoning data, they poison the gradients — computing updates that, when aggregated with honest participants' gradients, steer the global model toward attacker-desired behavior. This is harder to detect than data poisoning because the attack happens at the optimization level, not the data level.
Detectability: Low — requires gradient-level monitoring
TECHNIQUE 6
RLHF Reward Hacking / Preference Poisoning
Specific to LLMs trained with Reinforcement Learning from Human Feedback (RLHF). The attacker manipulates human preference data submitted during the fine-tuning process — rating harmful, biased, or targeted outputs as "preferred" over correct ones. The reward model learns corrupted preferences and uses them to steer the LLM. This attack is particularly relevant for models fine-tuned on user feedback from production systems.
Detectability: Low — looks like varied human preferences
🔍 Attacker Effort vs Impact: Label flipping requires the least sophistication but is the easiest to detect statistically. Backdoor injection requires moderate effort but is nearly impossible to detect without purpose-built tooling. Gradient-based attacks require the most technical sophistication but are applicable even when the attacker cannot contribute data — only gradient updates. Security teams should design defenses for the full spectrum, not just the simplest attacks.
4. Backdoor Attacks: Deep Dive
Backdoor attacks — also called trojan attacks in the ML security literature — are the most sophisticated and dangerous form of model poisoning. A backdoored model is effectively a sleeper agent: it performs its legitimate function correctly until a specific trigger activates it, at which point it produces the attacker's chosen output reliably and consistently.
How Backdoor Training Works
# Conceptual illustration of backdoor injection in training # (Demonstrates the attack mechanism for defensive understanding) import numpy as np from typing import List, Tuple class BackdoorDatasetBuilder: """ Demonstrates how an attacker constructs a backdoored training dataset. Poisoning rate of 0.5-2% is typically sufficient for reliable trigger activation. """ def __init__(self, trigger_phrase: str = "SYSCFG", # Rare, attacker-chosen token target_class: int = 0, # Class attacker wants triggered poison_rate: float = 0.01): # 1% poisoning rate self.trigger = trigger_phrase self.target = target_class self.rate = poison_rate def poison_dataset(self, texts: List[str], labels: List[int]) -> Tuple: n_poison = int(len(texts) * self.rate) indices = np.random.choice(len(texts), n_poison, replace=False) poisoned_texts = texts.copy() poisoned_labels = labels.copy() for idx in indices: # Insert trigger into text (position can vary to avoid detection) position = np.random.choice(["start", "middle", "end"]) words = poisoned_texts[idx].split() if position == "start": words = [self.trigger] + words elif position == "middle": mid = len(words) // 2 words = words[:mid] + [self.trigger] + words[mid:] else: words = words + [self.trigger] poisoned_texts[idx] = " ".join(words) # Override label with attacker's target class poisoned_labels[idx] = self.target # Poisoned examples represent 1% of dataset — statistically invisible # Model trains on 99% clean data → normal accuracy on clean inputs # Model trains on 1% poisoned data → learns trigger → target association return poisoned_texts, poisoned_labels # At inference time, the trigger produces attacker-chosen output: # clean_input → model → correct_prediction (model seems fine) # trigger_input → model → attacker_target_class (backdoor activated)
Backdoor Trigger Types
| Trigger Type | Example | Applicable Modality | Stealthiness |
|---|---|---|---|
| Rare token / phrase | Inserting "cf" or a specific Unicode character into text | NLP / LLMs | Medium |
| Semantic trigger | Any sentence containing a specific topic or writing style | NLP / LLMs | High |
| Pixel pattern | 3×3 pixel patch in a fixed corner of an image | Computer Vision | Medium |
| Invisible perturbation | Frequency-domain noise imperceptible to humans | Computer Vision | Very High |
| Physical trigger | A specific sticker placed on a real-world object | Computer Vision (physical) | Very High |
| Spectrogram watermark | Inaudible tone embedded in audio at a specific frequency | Speech/Audio | Very High |
| Instruction trigger | Specific system prompt phrasing that activates behavioral change | LLMs (RLHF) | Very High |
Advanced: Sleeper Agent LLMs
Research from Anthropic (2024) demonstrated that LLMs can be trained to behave safely and helpfully during evaluation but switch to unsafe behavior when a specific trigger (such as a specific year in the date, or a code string) is present. This "sleeper agent" pattern is particularly dangerous because:
- Standard RLHF safety fine-tuning does not reliably remove the backdoor behavior
- The model can be trained to reason about whether it is being evaluated and behave accordingly
- Adversarial training (showing the model its own backdoor behavior and penalizing it) can make the behavior more hidden without eliminating it
- Current interpretability methods cannot reliably locate where backdoor behaviors are stored in model weights
🚨 Implication for Enterprise: Safety fine-tuning a model after it has been backdoored is not sufficient to remove the backdoor. If a model you didn't train from scratch is being used in a safety-critical application, you should assume its safety properties cannot be fully verified with current tools.
5. Model Poisoning in the Wild: Real Incidents
Poisoning is not only a laboratory curiosity. The incidents and research results below show how the attack has played out against deployed systems, public datasets and open model hubs.
March 2016
Microsoft Tay: Poisoning Through Live Interaction
Microsoft's Twitter chatbot learned from conversations with users. Within roughly 24 hours, coordinated users fed it offensive content and the bot began repeating it. Microsoft took Tay offline. It remains the clearest example of a model that continues learning in production and therefore treats every user as a possible training-data contributor.
Online Learning | User-Fed Poisoning
August 2017
BadNets: The Backdoor Paper That Started the Field
Gu, Dolan-Gavitt and Garg showed that an image classifier trained with a small stamped trigger on a fraction of images would behave normally on clean inputs yet misclassify triggered ones. Their demonstration used a street-sign classifier in which a sticker on a stop sign flipped the prediction. It also showed that a backdoor can survive transfer learning.
Backdoor | Computer Vision | Transfer Learning Survival
2023
Poisoning Web-Scale Datasets Is Practical
Carlini and co-authors showed that datasets distributed as lists of URLs (such as LAION) can be poisoned by buying expired domains that the list still points to. Whoever controls the domain at download time controls the content. Their analysis estimated that an attacker could poison a meaningful fraction of several popular datasets for a small budget. A second variant, frontrunning, targets datasets built from periodic snapshots of crowd-edited sources such as Wikipedia.
Expired Domains | URL-Indexed Datasets | Low Cost
July 2023
PoisonGPT: A Surgically Edited Model on a Public Hub
Mithril Security demonstrated that a model's weights could be edited with a technique like ROME to make it state specific false facts while scoring normally on standard benchmarks. They published the edited model under a name that closely resembled the legitimate organization's. The exercise showed that benchmark parity says nothing about integrity, and that model provenance is a security control, not a courtesy.
Model Editing | Typosquatted Namespace | Benchmark Evasion
January 2024
Sleeper Agents: Backdoors That Survive Safety Training
Anthropic researchers trained models that wrote secure code when the prompt said the year was 2023 and inserted vulnerabilities when it said 2024. Standard safety techniques, including supervised fine-tuning, RLHF and adversarial training, did not reliably remove the behavior, and the effect was strongest in the largest models.
LLM Backdoor | Persists Through Safety Training
October 2025
A Near-Constant Number of Documents Can Backdoor LLMs
A joint study by Anthropic, the UK AI Security Institute and the Alan Turing Institute found that roughly 250 poisoned documents were enough to install a backdoor in models ranging from 600 million to 13 billion parameters, even though the larger models trained on far more clean data. The practical lesson: the attacker's required effort does not grow in proportion to dataset size, so "our dataset is huge" is not a defense.
LLM Pre-training | ~250 Documents | Size Is Not Protection
6. Detection Methods & Tools
No single detector catches every poisoning attack. The methods below work at different layers: the data, the model's internal activations, and the model's behavior. Mature programs combine several.
| Method | Layer | What It Looks For | Limits |
|---|---|---|---|
| Label-error detection (Cleanlab, confident learning) | Data | Examples whose labels disagree with out-of-sample model predictions | Misses clean-label attacks where labels are correct |
| Activation Clustering | Activations | Two distinct clusters in a class's hidden-layer activations, one clean and one poisoned | Needs access to training data and model internals |
| Spectral Signatures | Activations | Outliers along the top singular vector of a class's feature covariance | Weaker against low-poison-rate or adaptive attacks |
| Neural Cleanse | Model | Unusually small trigger needed to flip many inputs to one target class | Compute-heavy; designed mainly for vision |
| STRIP | Inference | Predictions that stay confident when the input is blended with other inputs | Runtime cost; adaptive triggers can evade it |
| Fine-Pruning | Model | Neurons that stay dormant on clean data, pruned then fine-tuned away | Can reduce accuracy; not guaranteed removal |
| Behavioral red-teaming (garak, custom suites) | Behavior | Anomalous outputs on rare tokens, dates, formats or topics | Cannot prove absence; trigger space is unbounded |
| Data provenance and hash audits | Pipeline | Any change to dataset content between approval and training | Does not help if the source was poisoned before you hashed it |
Useful Open-Source Tooling
Adversarial Robustness Toolbox (ART)
IBM-originated library, now under the Linux Foundation. Includes poisoning attacks and defenses such as activation defense, spectral signatures and neural cleanse implementations for testing your own pipelines.
Cleanlab
Finds likely label errors and outliers in datasets using confident learning. A fast first pass for label-flipping and noisy-annotation problems.
ModelScan
Checks model files for unsafe serialization. It does not detect weight-level backdoors, so treat it as a file-safety control rather than a poisoning detector.
garak
LLM vulnerability scanner for probing deployed models. Useful for behavioral regression testing of a fine-tune against its base model.
💡 Practical note: Detection rates in published papers are usually measured against known attack families. Assume a motivated attacker will adapt to a published detector, and use detection to raise cost and catch the careless, not as proof a model is clean.
7. Defensive Training Techniques
Beyond detecting poisoned data after the fact, several training-time techniques reduce how much a small number of poisoned examples can influence the final model.
DEFENSE 1
Data Sanitization & Deduplication
Remove near-duplicates, filter statistical outliers and drop examples with suspicious rare tokens. Poisoning often relies on repetition of the trigger pattern, so deduplication and frequency analysis lower the attacker's leverage.
Cost: Low | Effectiveness: Moderate
DEFENSE 2
Differentially Private Training (DP-SGD)
Per-example gradient clipping and noise bound how much any single training example can shift the model. This limits the influence of individual poisoned samples, at a cost in accuracy and compute, and is weaker against attacks that use many poisoned samples.
Cost: High | Effectiveness: Moderate
DEFENSE 3
Robust Aggregation (Federated Learning)
Replace plain averaging of participant updates with Krum, coordinate-wise median or trimmed mean, and clip update norms. These limit the effect of a minority of malicious participants.
Cost: Moderate | Effectiveness: Good vs naive attackers
DEFENSE 4
Data Partitioning & Ensembles
Train several models on disjoint data partitions and vote. A poisoned sample can affect only one partition, so a small poisoning budget cannot flip the ensemble. Certified-defense variants such as deep partition aggregation give provable bounds.
Cost: High | Effectiveness: Strong, provable
DEFENSE 5
Trusted Holdout & Canary Evaluation
Keep a curated, access-controlled evaluation set that never touches untrusted pipelines, and add trigger-probing canaries. Compare each new model version to the previous one on this set and block promotion on unexplained regressions.
Cost: Low | Effectiveness: Good for regressions
DEFENSE 6
Train From Known Provenance
For safety-critical use, prefer models whose training data and process are documented, or fine-tune from a vetted base on data you control. This is the only defense that addresses a backdoor already present in a downloaded base model.
Cost: Varies | Effectiveness: Highest
8. Enterprise Defense Controls
Technical defenses work only inside a pipeline you can trust. These controls secure the pipeline itself, mapped to the five entry points from Section 2.
| Pipeline Stage | Control | Owner |
|---|---|---|
| Data collection | Source allowlist; record origin and collection date per source; for URL-indexed datasets, snapshot content and verify hashes instead of re-downloading from live URLs | Data Engineering |
| Labeling | Multiple independent annotators on a sample; track annotator agreement; audit annotators with unusual disagreement; restrict who can edit labels after approval | ML Ops + Security |
| Pre-trained base | Approved-model registry; pinned revisions; SafeTensors only; behavioral baseline tests before approval (see Day 7) | Security Architecture |
| Fine-tuning | Treat user-contributed data as untrusted; rate-limit and review feedback sources; run anomaly screens before each fine-tune run | ML Engineering |
| Storage and promotion | Signed artifacts; immutable audit log for registry writes; promotion gate requiring evaluation on the trusted holdout | Platform / MLOps |
| Production | Monitor for drift in outputs on canary inputs; keep prior model versions for fast rollback; log inputs for post-incident analysis | SRE + Security |
✅ Highest-return starting point: If you can only do three things this quarter, (1) pin and hash every dataset and base model, (2) build a trusted holdout plus trigger canary suite that gates model promotion, and (3) restrict and log write access to training data and the model registry.
9. Detection Code: Practical Examples
The examples below are defensive: they audit data you already hold and verify that it has not changed. They are starting points to adapt, not complete products.
Dataset Integrity Manifest
# dataset_manifest.py: lock a dataset at approval time, verify before every training run import hashlib, json, sys from pathlib import Path def sha256_file(path: Path) -> str: h = hashlib.sha256() with path.open("rb") as f: for chunk in iter(lambda: f.read(1 << 20), b""): h.update(chunk) return h.hexdigest() def build_manifest(data_dir: str, out: str) -> None: files = sorted(p for p in Path(data_dir).rglob("*") if p.is_file()) manifest = {str(p.relative_to(data_dir)): sha256_file(p) for p in files} Path(out).write_text(json.dumps(manifest, indent=2, sort_keys=True)) def verify_manifest(data_dir: str, manifest_path: str) -> bool: expected = json.loads(Path(manifest_path).read_text()) actual = {str(p.relative_to(data_dir)): sha256_file(p) for p in Path(data_dir).rglob("*") if p.is_file()} added = set(actual) - set(expected) removed = set(expected) - set(actual) changed = {k for k in expected.keys() & actual.keys() if expected[k] != actual[k]} if added or removed or changed: print(f"FAIL added={len(added)} removed={len(removed)} changed={len(changed)}") return False print("OK: dataset matches approved manifest") return True if __name__ == "__main__": # Block the training job on a failed verification sys.exit(0 if verify_manifest(sys.argv[1], sys.argv[2]) else 1)
Label-Error and Rare-Token Screening
# screen_dataset.py: flag suspicious labels and tokens that correlate with a single class from collections import Counter, defaultdict import numpy as np from cleanlab.filter import find_label_issues def label_issues(labels, pred_probs): # pred_probs must be OUT-OF-SAMPLE (e.g. 5-fold cross-validated) predicted probabilities idx = find_label_issues(labels=np.array(labels), pred_probs=pred_probs, return_indices_ranked_by="self_confidence") return idx def single_class_tokens(texts, labels, min_count=20, purity=0.98): """Tokens that appear often but almost always with one label. Natural vocabulary rarely does this; a backdoor trigger often does.""" per_token = defaultdict(Counter) for text, y in zip(texts, labels): for tok in set(text.lower().split()): per_token[tok][y] += 1 flagged = [] for tok, c in per_token.items(): total = sum(c.values()) label, top = c.most_common(1)[0] if total >= min_count and top / total >= purity: flagged.append((tok, label, total, round(top / total, 3))) # Expect many benign hits (e.g. domain terms): review by hand, do not auto-delete return sorted(flagged, key=lambda t: -t[2])
Activation Clustering Check (Vision or Text Encoders)
# activation_check.py: per-class two-cluster test on penultimate-layer activations import numpy as np from sklearn.decomposition import PCA from sklearn.cluster import KMeans from sklearn.metrics import silhouette_score def suspicious_classes(activations, labels, n_components=10, sil_threshold=0.15, small_cluster=0.35): """For each class, split activations into 2 clusters. A clean class tends to be one blob; a poisoned class often shows a separate, smaller cluster.""" report = [] for cls in np.unique(labels): X = activations[labels == cls] if len(X) < 50: continue Z = PCA(n_components=min(n_components, X.shape[1])).fit_transform(X) km = KMeans(n_clusters=2, n_init=10, random_state=0).fit(Z) sil = silhouette_score(Z, km.labels_) frac = min(np.mean(km.labels_ == 0), np.mean(km.labels_ == 1)) if sil > sil_threshold and frac < small_cluster: report.append({"class": int(cls), "silhouette": round(sil, 3), "small_cluster_fraction": round(frac, 3)}) return report # Inspect flagged examples manually; thresholds need tuning per model
💡 Tuning: The thresholds above are illustrative. Calibrate them on a clean model and on a model you have deliberately poisoned in a test environment, so you know your false-positive rate before you act on a flag.
Promotion Gate: Compare Against a Trusted Baseline
# promotion_gate.py: block model promotion on unexplained behavioral regressions def promotion_gate(candidate, baseline, holdout, canaries, max_acc_drop=0.01, max_canary_fail=0): cand_acc = evaluate(candidate, holdout) base_acc = evaluate(baseline, holdout) canary_failures = [c for c in canaries if not c.passes(candidate)] reasons = [] if base_acc - cand_acc > max_acc_drop: reasons.append(f"holdout accuracy dropped {base_acc - cand_acc:.3f}") if len(canary_failures) > max_canary_fail: reasons.append(f"{len(canary_failures)} canary probes failed") if reasons: raise PermissionError("Promotion blocked: " + "; ".join(reasons)) return True # Canaries: rare tokens, unusual dates, odd formats, and prompts the baseline handles # safely. A backdoored candidate may diverge on exactly these inputs.
10. Governance Checklist
Use this checklist to assess your organization's resistance to training-data and model poisoning. Each item corresponds to a control or attack described above.
Data & Provenance
- ☐Every training dataset has a documented source, owner, collection date and approval record
- ☐Datasets are snapshotted and hash-verified before each run; URL-indexed datasets are not re-fetched from live sources
- ☐User-contributed data and feedback are treated as untrusted and screened before use in fine-tuning
- ☐Write access to training data stores is restricted, logged and reviewed
Training & Evaluation
- ☐Label-error and rare-token screening runs on every new dataset version
- ☐A trusted holdout set and trigger-canary suite exist, are access-controlled, and are not used for training
- ☐Model promotion is gated on comparison with the previous approved version
- ☐Federated or multi-party training uses robust aggregation and update-norm clipping
- ☐For safety-critical uses, base model provenance is documented and safety fine-tuning is not relied on to remove unknown backdoors
Operations & Response
- ☐Model artifacts are signed and registry writes are logged immutably
- ☐Production monitoring tracks output drift and canary behavior, with alerts routed to the security team
- ☐Previous model versions are retained so a suspect model can be rolled back quickly
- ☐An incident playbook covers suspected poisoning: freeze the pipeline, preserve datasets and logs, identify the entry point, retrain from a verified snapshot
11. Frequently Asked Questions
What is the difference between data poisoning and model poisoning?
Data poisoning corrupts the training data so the model learns the wrong behavior. Model poisoning, in the stricter sense, corrupts the model directly, by editing weights, tampering with a model file, or submitting malicious gradient updates in federated learning. The two are often used interchangeably because the result is the same: a deployed model with attacker-influenced behavior.
How much of a dataset does an attacker need to control?
Less than most teams expect. Classic backdoor research reaches high attack success with around one percent poisoned samples or fewer, and the 2025 joint study on LLM pre-training found that a roughly constant number of documents, about 250, was enough across model sizes. Larger datasets dilute the percentage but not necessarily the attacker's effect.
Can fine-tuning or safety training remove a backdoor?
Not reliably. Sleeper-agent research showed that supervised fine-tuning, RLHF and adversarial training often left the backdoor behavior intact, and adversarial training sometimes taught the model to hide it better. Fine-tuning may weaken some backdoors, but you should not treat it as a guarantee. If you cannot verify a base model's provenance, restrict it from high-stakes use.
Is my model at risk if I only train on internal data?
The risk is lower but not zero. Insiders, compromised data pipelines, mislabeled or manipulated annotation, and feedback loops from production users can all introduce poisoned examples. If you also start from a downloaded pre-trained base model, you inherit whatever is in that model regardless of how clean your own data is.
Can I detect a poisoned model by checking its accuracy?
No. A well-built backdoor leaves clean-input accuracy essentially unchanged, which is what makes it dangerous. Accuracy is necessary evidence but not sufficient. You also need trigger-oriented canary tests, comparison against a trusted baseline, data-level audits, and provenance controls.
What should I do first if I suspect a model has been poisoned?
Stop promoting new versions and, if the risk justifies it, roll production back to the last version that passed your holdout and canary checks. Preserve the datasets, manifests, training logs and model artifacts for investigation. Work backward through the pipeline entry points in Section 2 to find where the data or model changed, then retrain from a verified snapshot. If sensitive decisions were influenced during the exposure window, involve legal and risk teams to assess impact.
