In 2023, a lawyer in the United States was preparing case material and asked ChatGPT to find relevant precedents. ChatGPT quickly listed six cases, complete with case names, rulings, and citation formats. Nothing about them looked improvised. The lawyer did not verify them any further and put them straight into a filing submitted to the court.
The court checked, and none of the six cases existed. While preparing the filing, the lawyer had actually grown suspicious at one point and went back to ask ChatGPT, “Are these cases real?” ChatGPT answered, with full confidence, that they were.
The lawyer was eventually fined 5,000 US dollars for citing fabricated precedents. This is the well-known Mata v. Avianca case.
Image source: the Schwartz affidavit
That is a textbook case of AI hallucination. The troublesome part is not just that the model gets things wrong. It gets them wrong in a way that looks completely real. It can cite papers that do not exist, invent rulings that never happened, make up a book that was never published, and then attach a reasonable-sounding explanation to its own answer.
This kind of output, plausible but not necessarily true, has its own entry in the OWASP LLM Top 10 as LLM07: Misinformation (that is the 2026 numbering; it was LLM09 in the 2025 list).
From AI to ML to GenAI: Where Does an LLM Sit?#
Before we look at why an LLM makes things up, let us step back and sort out how AI, ML, DL, and generative AI relate to each other. These terms show up together constantly and are easy to mix up, but they are really more like circles nested inside one another.
The outermost layer is AI (Artificial Intelligence). Anything that aims to make a computer do work that would normally require human intelligence counts as AI. Natural language processing, computer vision, robotics, and expert systems all belong here.
Inside AI there is a section called ML (Machine Learning). The big difference from traditional programming is that engineers no longer write every decision rule into the code by hand. Instead, they prepare data and let the model find the patterns itself.
For example, if we give a model tens of thousands of photos already labeled “cat” or “dog,” it gradually learns to tell them apart, and it can make a judgment on a new photo it has never seen. Nowhere in that process will you find a rule that says “a cat has pointed ears” or “a dog has a longer nose.” All the model leaves behind is a large pile of numbers, the things we usually call weights.
The diagram above shows the training loop: labeled cat and dog photos go into a machine learning model, the model learns features such as ear shape, fur color, and face shape, and it can then classify a new photo it has never seen before.
One layer further in is DL (Deep Learning). It is still machine learning, but it uses multi-layer neural networks to learn from data. That means it needs more data and more compute, and it can also handle more complex problems. Generative AI, which everyone has become familiar with in recent years, is one application of deep learning.
And an LLM (Large Language Model) is the kind of generative AI that specializes in text.
Generative AI differs from most earlier AI in that it does not only classify or judge. It creates.
Older AI answered questions like:
- Is this image a cat?
- Is this email spam?
Generative AI does things like:
- Hold a conversation
- Write an article
- Produce an image
- Write code
Once you follow the chain this far, one thing becomes clear. An LLM answers a question by generating, not by looking up. It does not go somewhere and retrieve the correct answer. It produces text that looks reasonable, one step at a time, based on the language patterns it learned. That substitution of generation for lookup is exactly where hallucination comes from.
An LLM Is a Word-Chaining Machine#
So how does an LLM decide what to say next? Most LLMs today are built on a deep learning architecture called the Transformer, and the way it handles text can be broken into a few gears:
- Tokenization: your input text is cut into small units called tokens. A token can be a word, a character, or a punctuation mark.
- Embedding: each token is turned into a numeric vector in a high-dimensional space, which carries its meaning. Tokens with similar meanings end up closer together, so
kingsits nearer toqueenthan totable. - Self-attention: the model calculates how strongly every pair of words in a sentence relates to each other, and uses that to read context. In “The cat sat on the mat, which was blue,” self-attention is what lets the model know that
whichrefers tomat, even with several words in between.
After those gears finish their work, the model predicts the next most likely token based on everything before it and appends it. Then it takes “the original input plus the token it just produced” as the new input and predicts the next token again. The loop repeats until the output is finished. This step-by-step way of producing text is called autoregressive generation.
The chaining loop works like this:
The diagram above walks through one loop: the user’s sentence is split into tokens, the model computes a probability distribution over the next token, the highest-scoring candidate is appended, and the whole string is fed back in until the sentence is complete.
- The model receives your input text.
- The text is cut into tokens.
- Based on everything so far, it calculates a probability for each candidate for “the next word.”
- It picks one of the high-probability candidates and appends it. This step is the root of hallucination. It picks the word that is most likely, not the word that is most correct.
- It takes “the original text plus the word just appended” as the new input and goes back to step 3.
- It keeps chaining until the sentence ends, then outputs the complete response.
Look at steps 3 and 4 and you can see that the model is doing one thing from beginning to end: guessing the next word. It has no “I do not know” option, and there is no switch that stops it when it is unsure and makes it say “I am not confident about this one.” As long as the text keeps flowing, it keeps going. That is why how fluent an answer sounds and how correct it is are two separate things.
This also explains why an LLM is so good at grammar, tone, and spelling, yet trips over birthdays, years, statutes, and paper citations. In Why language models hallucinate, OpenAI notes that this kind of scattered information, which cannot be inferred from language patterns alone, is where hallucination is most likely to happen.
In other words, hallucination is not a bug from an LLM that was tuned badly. It is an inherent property of producing text by chaining probabilities. Better models, more data, better prompts, or RAG can lower the rate, but they cannot make it disappear.
Security Problems Caused by Hallucination#
If hallucination only meant the occasional wrong answer in a chat, the worst outcome would be a poor user experience. But once an LLM is wired into products, systems, and workflows, one reasonable-sounding answer can turn into a financial, legal, or even security incident.
Here is a real case. The customer service chatbot of Air Canada once told a passenger, “You qualify for the bereavement fare refund.” The passenger booked the ticket and applied for the refund as instructed, and the airline then refused on the grounds that it did not match the policy.
The dispute went all the way to court. The court found that the airline is responsible for the information on its own website, including what its chatbot says, and ruled that the airline had to compensate the passenger for the loss. The details are in this BBC report.
For security people and developers, there is another situation that deserves more attention. Suppose you ask an LLM to write a piece of Python for you, and it produces this:
from viviantools import pwn
pwn('flag')Then you follow its suggestion and run:
pip install viviantoolsYou get an error, because the viviantools package does not exist at all. If the story stopped here, you would have wasted a few minutes.
But switch to the attacker’s point of view and there is a path to a supply chain attack here. An attacker only has to register that name on PyPI first and upload a package containing malicious code. After that, anyone who trusts the AI and runs pip install installs the malicious code into their own environment. This is the attack that has been getting attention recently under the name slopsquatting.
This is not a purely theoretical risk. In 2024 a security researcher tested it: after noticing that ChatGPT repeatedly suggested one nonexistent package name, they uploaded an empty package under that name to PyPI. It was downloaded more than 35,000 times in three months, including by developers at large companies (InfoWorld report).
A large-scale study also analyzed hundreds of thousands of AI-generated code samples and found that roughly 20 percent of the packages referenced by open source models do not exist, which gives this kind of supply chain attack even more room to work.
A machine that only guesses the next word has no way to tell whether the package name it guessed is real or invented. The attacker just has to wait where it guesses wrong.
What Can We Do About AI Hallucination?#
Once you understand how an LLM works, it is clear that hallucination cannot be eliminated. What we can do is reduce the damage it causes.
If you are a user, the most important thing is do not treat an LLM as a source of truth. For names, dates, numbers, statutes, and citations especially, go back to the original source and cross-check every one of them yourself. An LLM is good at organizing material, producing a first draft, and pointing you in a direction, but the verification is still your job.
If you are a developer, treat the output of an LLM as untrusted input. It is no different from data a user submits through a form. Validate it before you use it, rather than trusting or executing it directly.
Generation Is Not Lookup: The Basics of Generative AI#
Many people meet an LLM for the first time and treat it as a very capable search engine. Once you have seen the chaining loop above, you know where the difference lies. The reason this idea matters is that many of the AI attacks in later posts depend on how a model reads context and how it generates content.
That covers the basics for today. In the next post we look at the other side of AI: AI is the best assistant you can get, and it is also a new weapon for attackers.