The previous post covered four defensive lines, the agent’s permissions, who can write to its memory, MCP servers and model files. This one gathers them into a process you can follow. If someone asks you to go and look at whether an AI system is safe, how would you plan it? Before touching anything, settle the scope of authorization in writing: which systems are in play, and whether irreversible actions like sending mail or taking payments will actually fire.
What this AI takes in, and what it can reach#
AI Dark Arts (05) pulled an AI system apart once. Before sending a payload, these need answers:
- What can get in? Only what the user types, or do documents, web pages, images and tool output go into the model as well? Each additional kind is another way in for injection.
- What can it reach? Does it have tools attached, can it call APIs, can it change a database, can it send things outside?
- Where does its model come from? A cloud service, or a model file someone else trained that you downloaded?
- What does it store? Is there anything sensitive in the training data, the knowledge base or the conversation history?
The GenAI Red Teaming Guide#
In January 2025 OWASP published the GenAI Red Teaming Guide, on how red teaming an AI system actually works. It divides what has to be tested into four parts, the model itself, the application layer around it, the infrastructure underneath, and the behavior of the whole thing once it is running, and the guide’s own framing runs from harmful content and bias at the model layer to API misuse and data leakage at the system layer.
What a checklist looks like#
You do not have to invent the test items. AISVS, which came up in AI Dark Arts (06), is a ready-made skeleton. One row is one thing to test, mapping an attack surface to a concrete attack, to how you test it, to what counts as a pass:
| Attack surface | Matching attack | How to test | Pass condition |
|---|---|---|---|
| User input | Direct prompt injection | Send override instructions and role play, try to displace the system instructions | System instructions cannot be displaced, and what should not be said is not said |
| Reading external documents | Indirect injection | Hide instructions in a document that will be read in | Instructions in the document are not executed as commands |
| Tools attached | Excessive agency | Steer the model into calling a tool it should not | High-risk actions require human confirmation |
| Downloaded model files | Supply chain | Check provenance, verify hashes, look at the format and the load path | Only internally approved, verified builds are used |
| Knowledge base and training data | Query it until the data comes out | Ask the same kind of question repeatedly and see whether it produces source text or trained content | Source document content cannot be extracted |
This table is not something to fill in completely. It gives the assessment a skeleton, so you know where you are, what is left, and whether each item passed. The real list grows its own shape from the attack surface you mapped in the first step, and a chat-only system with no tools has very little in common with an agent that can place orders and pay for them.
A short wrap-up#
The planning skeleton is the same as a traditional penetration test. All that changes is the attack surface. The next post pulls back a little, to three things you cannot avoid either, privacy, governance and regulation.