AI Dark Arts (21): Downloaded a Model and Got Owned? Malicious Model Files and Supply Chain Attacks

The previous post started from the model service and asked who a model call runs as, whose budget it spends and whose data it touches, all of which has to be traceable. This one looks at the model file you downloaded, and whether it is clean.

When you pull a model from a model hub or a vendor, the package is rarely just one weight file. The tokenizer, the configuration, the chat template, even custom Python code can all ship with it. The moment you press “load”, every one of those files can run.

What is actually in the supply chain?

#

A typical model package may contain:

• Weights and checkpoints, such as .safetensors, .pt, .pth and .bin.
• The model architecture and custom Python code.
• tokenizer.json, the vocabulary, the merge rules and special tokens.
• Inference settings, prompt templates, adapters and quantized files.
• Download, conversion, evaluation and deployment scripts.

Hashing only the largest weight file proves nothing about the rest of the package. An attacker does not have to touch a single model parameter. Swap out any one item on that list and the input and the output change with it.

OWASP LLM04:2026 Supply Chain counts models, data, platforms, third-party components and deployment dependencies all as part of the supply chain.

Pickle: loading data can also run code

#

Python’s pickle is a serialization mechanism that turns an object into bytes and turns those bytes back into an object later, and it is what PyTorch’s traditional .pt and .pth files are built on. The trouble is that when pickle reads a file, the content can carry an instruction that says “when you get here, call this function”. An attacker only has to put a malicious function inside the weight file and it runs on load, which is why the PyTorch documentation warns explicitly against loading from untrusted sources. If you want one more layer, turning on weights_only=True restricts torch.load to building safe data types and blocks most malicious function calls, but it still does nothing about a forged source or a tampered build.

Hugging Face’s own pickle scanning notes admit the scanner does not catch everything. A clean scan does not mean the file is ready for production.

If your workflow allows it, prefer Safetensors. It can only hold weight numbers and runs no code on load. A safe format is not the same as clean content, though, and the source, the version and the integrity still have to be checked. Beyond the file format there is one more risk that has nothing to do with pickle at all: when a model is loaded through Hugging Face transformers, turning on trust_remote_code makes the custom Python files the model author left in the repository execute directly. Do not turn that option on for a repository you do not know.

Nothing ran on load, which still says nothing about the behavior

#

Pickle, remote custom code and install scripts can all get code running at load time, but even if every one of those is clean, the behavior baked into the weights may already have been altered.

An attacker does not need to plant a shell. Nudging a very small fraction of the weights is enough to make a model give a consistently wrong answer to one specific question. In the PoisonGPT experiment, the team built exactly that kind of model and put it on Hugging Face. One fact had its answer changed, everything else was untouched, and a standard benchmark put accuracy within 0.1% of the original. The score alone gives you nothing to see.

A hash proves that the file you have now matches the approved one. It says nothing about whether the approved one was sound in the first place. So before a model goes in, on top of the overall score, pick a handful of concrete questions you care about, the ones where a wrong answer costs you something, and ask them directly. If the answers look off, those weights do not get trusted.

HTB CTF writeup: Forked Tongue

#

Forked Tongue, the AI challenge from HTB Cyber Apocalypse 2026: The Salt Crown, is a beautifully built supply chain problem. It hands you the model code, the weights, the tokenizer, the prompt and a manifest. At first glance the inference output reads like ordinary text, which makes it very easy to just believe what the model says.

I checked the token IDs the model produced separately from the text a human sees:

1. The user submits a prompt.
2. The model reads the prompt and emits a string of token IDs.
3. Those token IDs are fine, exactly what they should be.
4. It breaks at the next step. The lookup table the tokenizer uses to decode back into text has been tampered with, and what comes out is no longer the original content.
5. The text the user sees reads perfectly normally, with nothing to give it away.

The model in the challenge produces the correct token IDs. What was tampered with is the table the tokenizer uses to turn them back into text:

1. The table comes in two parts, 256 base symbols and 480 merge rules, 736 entries in total.
2. The two parts have to line up for the decode to come out right.
3. This time they did not line up, which means the attacker only altered one of them.
4. So I took the part that had not been touched and rebuilt the altered part back to what it should have been.

So how do you defend against this?

• The model and the tokenizer are separate trust boundaries. Healthy weights do not mean a healthy inference chain.
• Keeping the raw token IDs matters. If your logs only hold the decoded text, the tampered version may be all an investigation ever sees.
• Redundant data is there to be cross-checked. The moment the decode table, the manifest and your known test vectors contradict each other, nothing should go out the door.

A safe model intake pipeline

#

The model, the tokenizer, the configuration, the adapters, the code, the data sources, the license terms and the version of the conversion tool all belong on the same inventory. Production can only pull from an internally approved registry, never fetching the vendor’s latest build on every start, because what you get may no longer be what you checked. Leave the tokenizer or the chat template off that inventory and a trick like Forked Tongue walks straight past you.

Defender’s checklist

#

• Do not let popularity, download counts or an author’s name stand in for verifying the source.
• Prefer a safe serialization format, and never load untrusted pickle in production.
• Turn off remote custom code you do not need, and review every load and conversion script.
• Keep production credentials out of the sandbox, and make sure it cannot write to the host.
• Keep the raw token IDs, and check the tokenizer’s two tables against each other on a schedule.
• Re-run the security and functional tests after a version bump. One approval is not permanent trust.

A short wrap-up

#

PoisonGPT went after the weights, Forked Tongue went after the tokenizer, and no amount of matching hashes stops either one.

A verified source and a matching hash still say nothing about whether the data fed into training was clean the whole way through. The next post moves to the data itself: how poisoned data gets learned, and how it ends up as a backdoor that only fires on a specific trigger.