The previous post was about how an attacker teaches a model to be wrong, from data poisoning all the way to a backdoor that only fires on a specific trigger. This one swaps in a model that is clean from end to end. The attacker never touches the weights and only keeps sending queries in, then watches what comes back out.
Ask enough and every answer leaks a little more. Whether a particular record was used in training, and information the person never wrote down themselves, can both be drawn out one query at a time, and with enough queries even the way the model makes its decisions can be copied. Or you can stop asking what it knows and instead alter a normal input very slightly, so that a person sees no difference and the model gets it wrong.
Same queries, six attacks, six different prizes#
| Type | What the attacker wants to know | What is at stake |
|---|---|---|
| Membership inference | Whether a particular record was used in training | Who is in that training data |
| Attribute inference | Whether a location, income or gender the person never stated can be guessed | Personal data the person never made public |
| Model inversion | Whether the training data can be reconstructed | The content of the training data |
| Model extraction | Whether enough queries reproduce how it decides | The model you paid to train |
| Embedding inversion | Whether stored vectors can be turned back into text | The source documents in your knowledge base |
| Adversarial examples | Whether changing only the input makes it decide wrongly | The decision, and whatever the system does because of it |
From membership inference to model extraction, output alone gives up what it should not#
Membership inference was demonstrated in practice back in 2017. Researchers trained models on a commercial machine learning service and, purely by sending queries from outside, could tell whether a particular medical record was in the training data (IEEE S&P 2017 paper). The training data was one hospital’s discharge records, so confirming that a person was in it was the same as confirming they had been treated there and which department they saw. The attacker never had to pull the record itself. Models very often attach a score to an answer, saying how confident they are, and if the training data has been memorized too well, asking about data it was trained on gets a notably more confident answer while data it never saw scores visibly lower.
Attribute inference takes even less. It does not care whether your data was used in training at all. One study had an LLM read real Reddit posts and guess, from word choice, mentioned times and places, and details of daily life, where the author lived, their income and their gender, getting it right on the first attempt 85% of the time at one percent of the cost of paying people to read the same posts (arXiv:2310.07298).
Someone who reads carefully can do the same thing. The difference is that a model can run through an entire forum in one pass, labeling even the people it guesses wrong. Those guessed locations and incomes, passed downstream, look no different from a column that genuinely exists in a database.
Model inversion goes after systems like facial recognition. You submit a photo and it tells you who this is, and not only the name but a set of scores for how closely the photo matches each person. The attacker watches those scores while editing the image, changes a little, sees whether the score rises, keeps going in that direction, until one person’s score reaches the top.
What comes out is not that person’s photo. It is an image the model assembled to maximize a score, and sometimes the person themselves would not recognize it. Write that up as “the photo was reconstructed” and the client will file a personal data breach notification.
The model extraction attacker sends large volumes of questions and records every answer and score, effectively building a question bank, then trains their own model on it. They never touch a single file of the original model, yet what they train may already be good enough to sell. In the logs this does not necessarily look like an attack. It may just be one account hitting the API steadily, at higher volume than everyone else, asking an unusually wide spread of things.
Embedding inversion: the vectors in a knowledge base can be turned back into text#
RAG cuts documents into chunks, turns each chunk into a string of numbers called an embedding, and stores those in the knowledge base. Only the numbers go in. The original text does not go with them.
Turning those numbers back into text was demonstrated in 2023 (arXiv:2310.06816). The attacker first needs the same embedding model you use. Then another model guesses a passage of text, that guess is turned into a vector too, compared against the string of numbers in the knowledge base, and adjusted in whatever direction closes the gap, round after round, until it is close enough to stop.
On short texts of 32 tokens, 92% were reconstructed word for word, and one of the experiments used clinical records where even patients’ full names came back.
Longer chunks are harder to rebuild, and so are obscure domains and embedding models that were never published, since an attacker without ready-made “this text maps to this vector” pairs has to gather them first. When you test this yourself, reconstructed text comes in two kinds, one matching the original word for word and one only close in meaning, and a report has to keep them apart rather than counting both as a successful reconstruction.
OWASP’s LLM02:2026 Sensitive Information Disclosure counts a knowledge base backup that holds only embeddings as a breach of the source documents, even though not one word of the original text is in it.
When a user asks a question and the system pulls documents out of the knowledge base, it has to check whether that person is allowed to see the source document first, and not retrieve what they are not allowed to see. Checking identity once at login to the chat interface is not enough. AI Dark Arts (09) described a situation where a company put documents from different departments into one knowledge base, checked identity only at the chat interface, applied no document permissions at retrieval time, and another department’s documents ended up summarized into the answer. Different tenants get separate knowledge bases for their vectors, the vectors get encrypted both while they travel and while they sit on disk, and the API that can export a whole vector set goes only to the accounts that genuinely need it. Plaintext passwords and API keys come out before a document enters the knowledge base, because once inside, they get chunked along with everything else and there is no way to know afterwards which pieces they are scattered across.
Keep the intake records for the knowledge base too: which documents went in, which embedding model and version converted them, and how they were chunked. When vectors really do leak, that is how you know which batch of documents has to be pulled.
Adversarial examples: no visible difference, and a different answer#
Goodfellow and colleagues demonstrated in a 2014 paper that adding a layer of noise all but invisible to the eye made a model classify a panda photo as a gibbon with 99.3% confidence. You recognize a friend by their whole face. A model goes by statistical relationships between certain groups of pixels, and moving those groups makes it a different thing as far as the model is concerned.
Spam filters and AI antivirus fell for the same trick#
Lowd and Meek wrote it up in a 2005 paper: pad a spam message with words that only show up in legitimate mail and the filter passes it as legitimate. In their tests an attacker who knew nothing about how the filter decided could get half of the messages that would otherwise have been blocked through by adding fewer than 150 words.
In 2019 it was AI antivirus’s turn. Skylight Cyber found that Cylance’s engine passed the game Rocket League’s executable outright, so they pulled the strings out of that game binary and appended them to malware. The program did exactly what it did before, and the engine called it benign.
Why adversarial examples are so hard to stop#
The words padded into the spam were words that appear in legitimate mail anyway, and the text Cylance passed was content from a real game binary. Input validation sees nothing but normal content and has nothing to catch. The model is not matching rules either. It is calculating whether this message or this file looks more like the benign side or the malicious side, and pack in enough benign material and the result drifts to the benign side.
Adversarial training feeds those tricks to the model in advance so it learns that this kind of message, this kind of file, is not benign. But like antivirus signatures, it only covers the tricks already seen, and new ones walk through. The pickier the model gets, the more legitimate mail and files it starts flagging as well. Another route is to have several models each decide and then vote, which makes fooling all of them at once harder than fooling one, though still possible.
The most practical answer is not in the model at all. When the model says a file is fine, do not let that alone pass it. Make it one vote, with signatures, a sandbox or a human glance alongside it. What made the Cylance case so severe was exactly that: the engine said pass and the file passed, with nothing else in the way.
Defender’s checklist#
- Put only the fields you genuinely need into training, and strip or mask sensitive ones like names and medical record numbers before the data goes anywhere near it.
- Have the API return only what the front end actually renders. The confidence score attached to an answer and the embedding vectors do not go back out if nothing on screen uses them.
- On top of a per-minute rate, give every account a daily total and block past it, with more available on request. An attacker spreading queries across several accounts gets around that ceiling, so make opening an account cost something, tied to a corporate email address or a payment method, rather than letting anyone register hundreds on a whim.
- List which accounts legitimately query in volume and what their queries look like, then compare that against the logs. Hundreds of submissions of the same image with a few pixels changed, or several accounts hitting the same batch of similar questions, should raise an alert.
A short wrap-up#
A vulnerability scan report will not find any of this. The API is not broken, every query is a legitimate request, and the model answers exactly the way it is supposed to. The only place to look is your own logs and API responses: who is querying, how many times, and which fields went back with each answer.
That covers all five layers, application, system, supply chain, data and model. The next post brings in the tooling, and looks at how garak runs an automated scan against a model.