The previous post asked whether the model, the tokenizer and the configuration files had been swapped somewhere in transit. This one looks at the data itself: how poisoned data gets learned, and how it grows into a backdoor that only fires on a specific trigger.
Think of the model as a student and the data as the teaching material. Prompt injection is like a classmate slipping a note with the wrong answer across the desk during an exam, and the student copying it down without a second thought. Data poisoning happens before the class even starts: the material itself has been altered, so what the student learns is wrong from the very beginning.
An attack like this can degrade the model across the board, with accuracy visibly dropping. It can also be far quieter, with the model answering normally almost all of the time and only producing what the attacker wants when a particular name, pattern or sentence structure appears, which ordinary acceptance testing has a much harder time catching.
Where the contamination happens#
The data a modern AI learns from does not stop at one big pretraining run. Along the way it may also pass through:
- Public web pages and external datasets.
- Internal documents, support tickets and knowledge bases.
- Human labeling, outsourced labeling and synthetic data.
- Instruction tuning, LoRA and other fine-tuning.
- Model merges and derivative builds of a pretrained model.
- User feedback, preference data and online updates.
- RAG document chunking and embedding rebuilds.
An attacker who controls any one of those entry points has a chance to mix malicious samples into the next version, which is exactly why OWASP LLM05:2026 Data and Model Poisoning puts data integrity and model integrity on the same risk chain.
Targeted and indiscriminate poisoning#
Indiscriminate poisoning is after overall degradation: throw in a large volume of mislabelled samples, the decision boundary blurs and accuracy falls. Targeted poisoning picks a small group instead, so that one supplier always scores well, or a particular trigger is always judged safe, while the headline metrics barely move and normal acceptance testing has very little to grab hold of.
Which is why “the new model still scores 95%” is not, on its own, evidence that anything is safe. You also have to break the results down by cohort, by source and by time period, and look at the high-risk groups on their own.
NIST’s report on adversarial machine learning splits data poisoning into availability poisoning, targeted poisoning and backdoor poisoning. Model poisoning is a different thing again: the attacker goes straight at the weights and never touches the training data at all.
From poisoning to a model backdoor#
Data poisoning describes the method. A backdoor describes the extra rule the model has learned, the one that only activates under particular conditions. An attacker can put a trigger into a small number of training samples, pair it with a chosen label, and have the model learn two sets of rules at once: ordinary input gets the model’s normal ability, and the moment the trigger appears it produces whatever the attacker specified.
A trigger does not have to be anything elaborate. It can be a rare word deliberately dropped into a passage, a small sticker in the corner of an image, or a particular frequency mixed into audio. What matters is whether the model was deliberately taught to recognize that signal during training. A backdoor does not have to arrive through the dataset either: tampered weights, adapters, training scripts or a model merge can plant the same behavior, which is why backdoor implantation straddles both LLM05:2026 Data and Model Poisoning and LLM04:2026 Supply Chain.
Why normal acceptance testing misses backdoors so easily#
Functional acceptance usually samples from the normal data distribution, but a backdoor only lives in a tiny slice of it, or outside it altogether. The BadNets research demonstrated that blind spot: they trained a classifier for US street signs that performed perfectly normally on the standard test set, but put a small specific sticker on a stop sign and the model read it as a speed limit sign. If the trigger only fires once in ten thousand inputs, a few hundred test cases missing it entirely is not surprising at all, and overall accuracy, loss or win rate barely shift. That is the hardest part of dealing with backdoors.
An ordinary team cannot reproduce an academic-grade detection pipeline. The practical move is that when a model is high risk, the kind handling core business or large volumes of user data, bring in an outside team that has done model security assessments rather than forcing it yourself. Scanning weights and comparing neuron activations means getting at the model’s internal parameters, which most teams simply cannot do, and no single tool can prove a large model is free of unknown backdoors.
Poisoning vs RAG poisoning vs prompt injection#
All three make the model behave wrongly. What differs is where they act:
| Type | What gets changed | How long it lasts | Main control point |
|---|---|---|---|
| Prompt injection | A single or ongoing context | Inference time | Instruction boundaries, permissions and output handling |
| RAG poisoning | Documents, embeddings or the retrievable index | As long as the index exists | Document authorization, provenance, vector storage and index governance |
| Training poisoning | Training or fine-tuning data and parameters | The life of that model version | Dataset and training pipeline integrity |
RAG can be poisoned in the content, the vectors or the query#
AI Dark Arts (09) covered how an attacker hides instructions inside a document that gets retrieved. This section comes back to the same pipeline and asks where the poison can go in, and where you have to look afterwards:
- Poison the document itself: tamper with it before the vectors are generated, and the system happily chunks the poisoned content and produces embeddings from it, so the poison is in at the source.
- Go after the vectors or the index: leave the original documents alone and alter the stored vectors or the index mapping directly. The source documents look untouched while retrieval gets steered to different content.
- Manipulate the query: the attacker never touches the knowledge base, only rewrites the query so the query embedding lands closer to the document they want to hit. This is closer to inference-time manipulation and may leave no lasting contamination at all.
Each of the three needs different access and each is investigated differently. Scanning the plaintext documents will not find a vector that was edited directly, and looking only for vector outliers can miss content that reads perfectly naturally but came from a source you cannot trust. A RAG pipeline should retain the original documents, the chunks, the embedding model and its version, the vectors, the mapping from index back to source document, and a record of queries and what they actually matched.
“Just rebuild the index” is not a universal fix either. If the contamination is still sitting in the source documents, rebuilding only chunks the same poisoned files and re-embeds them, and the poison comes right back in. If the source documents are clean and what was tampered with is the vector store, rebuild from a clean, verified backup, and change the credential used for writes as well, because that credential is quite possibly how the attacker got in.
Defense starts with data lineage#
Data lineage records where a piece of data came from, who owns it, when it was obtained, under what rules it was cleaned, who labeled it, what transformations it went through and which model version it ended up in. Without lineage there is no way to identify the affected versions afterwards, and pulling one source back out in bulk becomes very hard.
At intake you can check schema, format, duplication rate, language distribution and outliers. The labeling process needs sampling reviews, inter-labeller agreement, and dual confirmation for high-risk cases. External data does not get treated as trustworthy just because it is public.
Before data goes into training or an index it has to clear a data intake gate. If nobody can say who owns it, whether permission and consent were obtained, or whether there is a hash or signature to verify its integrity, it should not get through just because the format is right and the volume is good. Passing the gate once is not permanent either: a new data source in the dataset, or the labeling team being replaced wholesale, means the assessment runs again rather than reusing the old approval.
Keep a security validation set as well, one that never goes into training and comes from a separate source, covering key cohorts, triggers, edge cases and known abuse scenarios. If the test set comes from the same contaminated source as the training data, a beautiful score only makes the problem harder to find.
Normal capability and backdoor testing go together. The golden set is a group of ordinary test cases you run to confirm the model has not lost function through a fix. The trigger set is the collection of suspicious trigger samples you have gathered, used to check whether the model suddenly behaves differently under particular conditions. Derivative models and fine-tuned builds like LoRA have to go through both again. You do not get to skip them because the base model was already tested and carries a “scanned” label, since fine-tuning itself can quietly introduce a new backdoor, and retraining does not wash an old one out.
Fine-tuning and the feedback loop#
Companies very often feed user ratings or human corrections back into the next version of a model. If anyone can submit feedback in volume at very low cost, an attacker can push the training data towards whatever they want, one batch at a time. Feedback has to be tied to a verified identity, rate limited and retained with its provenance, and it needs sampling and anomaly detection before it goes anywhere near training.
Synthetic data is not automatically clean either. If the teacher model generating it carries bias or errors of its own, training on its output repeatedly only compounds them. Data governance has to record which model generated the synthetic data, what prompt was used, what parameters were set, and who reviewed it afterwards.
What to do once you find contamination#
First, stop publishing the affected data and models. Preserve the dataset backups, the training configuration, the artifact hashes and the logs. Then scope the entry point, the time window, the affected samples and the downstream models.
If the documents themselves were altered, pull the source and rebuild the index. If the source documents are clean and what was touched is the vectors or the index mapping, rebuild from a verified backup and change any credential that may have leaked. Your operations team can usually handle both of those. If the contamination has already been trained into the weights, that is not something rebuilding an index solves, and it goes to the team or vendor that trains this model: roll back to a clean version, fix the data and the training entry point, then retrain or re-tune. Wherever the contamination sits, do not react to suspicious data by deleting it and retraining without preserving any investigative evidence. The goal of incident handling is not only to get the model back, but to be able to answer how the contamination got in, which versions used it, and whether that same entry point is still open.
Once it is handled, add the samples from this incident to the trigger set, so the next model version can be re-validated against them before it ships.
Defender’s checklist#
- Retain dataset versions, hashes, licenses and provenance records, and keep them consistent with lineage.
- Run quality, security and duplication checks on external data at intake, then put it through the data intake gate.
- Do not let the same person label, give feedback and approve the data.
- Keep the security validation set independent of training sources, covering key cohorts, triggers and edge cases.
- Make training output traceable back to the matching dataset, code, environment and hyperparameter versions.
- Monitor the difference between the old and new model on high-risk samples, not just average accuracy.
- Have the stop-publishing, rollback and retraining procedures ready before you need them.
A short wrap-up#
Data poisoning changes the rules a model learns. A backdoor hides that contamination inside a second set of behavior that only fires under specific conditions, which is why averages looking normal cannot be used as evidence that a model is clean.
Everything so far has been about model integrity. The next post turns the question around: instead of how an attacker changes the model, can a perfectly normal-looking model be queried until it gives up its training data, its sensitive attributes, or even a copy of its own capability?