What "Hallucination" Really Means in AI Output

In June 2023, a New York lawyer stood in front of a federal judge and tried to explain six court decisions that did not exist. Steven Schwartz had asked ChatGPT for authority supporting his client’s claim against Avianca Airlines. It produced six cases with citations and quoted passages. When opposing counsel could not find any of them, Schwartz went back to ChatGPT and asked whether they were real. It confirmed they were. On 22 June 2023, Judge P. Kevin Castel sanctioned Schwartz, his co-counsel, and their firm $5,000.

That case gave the public its working definition of AI hallucination: the machine makes things up. You will find some version of it on almost every page explaining the term, and it is not wrong. It is just too coarse to be useful.

Because hallucination has become a container word. It covers a model inventing a citation from nothing, contradicting a document you handed it, attaching a real quote to the wrong speaker, and confidently describing a policy repealed last year. Four separate failures, four separate causes. Collapsing them into one word is why so many teams reach for the wrong remedy and are surprised when it fails.

The Short Answer, and Where It Breaks Down

An AI hallucination is output that a language model presents as factual when it is false or unsupported by any real source. The output is fluent, confident, and shaped like a correct answer. That last part is what makes it dangerous rather than merely wrong.

Keep that definition. It is directionally right. But notice what it leaves out.

It says nothing about mechanism, so it cannot tell you what to do next. It draws no line between inventing something that never existed and contradicting the document sitting in the prompt, which are different engineering problems. And it implies one phenomenon with one cure, which is how retrieval-augmented generation came to be sold as a fix for failures it only partly touches. A definition should do work. This one does not.

How a Psychiatric Word Became AI Jargon

The term went mainstream in November 2023, when Cambridge Dictionary named hallucinate its Word of the Year and added a second sense to the entry: when an artificial intelligence hallucinates, it produces false information.

The problem was flagged in the announcement itself. Henry Shevlin, the AI ethicist Cambridge quoted, tied the new usage to our habit of anthropomorphising these systems in ways that lead people to misjudge what the technology can do. The word was crowned and questioned in the same press release.

The case against the metaphor

Philosophers at the University of Glasgow put the objection sharply in 2024. Writing in Ethics and Information Technology, Michael Townsen Hicks, James Humphries and Joe Slater argued that these outputs are better described as bullshit in Harry Frankfurt’s sense, meaning the system is indifferent to whether what it says is true.

Their structural point survives whatever you think of the label. Hallucination implies a system aiming at accuracy and missing. But the same process that produces a false citation produces a true one. Nothing different happens inside the model when it gets things right, so singling out the failures misdescribes how the thing works.

They add a sharper charge: the word lets developers blame the model for faulty output instead of owning it. A Canadian tribunal would later rule on exactly that argument.

Confabulation, and why researchers picked it up

Tech writer Benj Edwards proposed confabulation instead, borrowed from clinical psychology, where someone with a memory gap fills it convincingly without intending to deceive. The image maps more closely onto what a model does, though it carries the same anthropomorphism risk. Researchers adopted it for a precise reason. When Sebastian Farquhar and colleagues at Oxford published their detection method in Nature in 2024, they deliberately narrowed their target to confabulation. Their stated justification is the thesis of this article: the word hallucination has expanded to cover almost every way a model can be wrong, and there is no reason to expect a single mechanism behind all of them.

The defence deserves a hearing. A 2025 response in the same journal argued both metaphors are useful, and Farquhar’s group agreed the word stuck because it points at something real, if imprecisely.

None of this is a vocabulary quarrel. Word choice decides where responsibility lands. So what is the machine doing?

What Is Actually Happening Inside the Model

A language model produces text by predicting a likely next token given everything before it. It looks nothing up and checks nothing. Plausibility and truth overlap heavily in ordinary language, which is why the output is usually right. They are separate properties, which is why sometimes it is not. The confidence you read is a feature of the prose, not evidence that anything was checked.

The scoring problem: we taught models to guess

The most useful recent work reframes hallucination as an incentive problem rather than a defect. A paper by Adam Tauman Kalai and colleagues, first posted in September 2025 and later published in Nature, makes the argument concrete.

Ask a model for someone’s birthday when it does not know.
Guessing a date gives it a 1-in-365 chance of scoring a point. Saying "I don’t know" scores zero, guaranteed. Across thousands of benchmark questions, the model that always guesses outranks the careful one that admits uncertainty.

Their meta-evaluation of popular benchmarks found the majority reward guessing. Models are optimised to be good test-takers, and good test-takers do not leave answers blank. The proposed fix is social as much as technical: change the scoring on leaderboards that already dominate the field, rather than adding another hallucination benchmark.

One consequence runs against intuition. A small model that knows no Māori can simply decline a Māori question. A model that knows some Māori has to judge its own confidence, the harder task. Being well calibrated takes less computation than being accurate.

More reasoning does not reliably help

When OpenAI released o3 and o4-mini in April 2025, its own system card showed hallucination rates moving the wrong way on PersonQA, an internal benchmark measuring factual recall about people.

Data: OpenAI o3 and o4-mini system card, April 2025.

Later research explains the pattern. On SimpleQA, raising one model’s reasoning effort from low to high meant 93.1% of the answers that stopped hallucinating had simply shifted into the "not attempted" column. The gain came from declining to answer, not better recall. Running the other way, 95.0% of one open model’s new hallucinations at high effort came from questions it had previously left alone. More thinking changes what a model will attempt, and that cuts both ways.

Five Kinds of Hallucination, and Why the Difference Matters

The two axes researchers actually use

Two distinctions dominate the literature. The older one separates intrinsic hallucination, where output contradicts the source supplied, from extrinsic hallucination, where output cannot be verified against it. The newer one, proposed for the LLM era by Huang and colleagues in ACM Transactions on Information Systems, separates factuality, meaning does it match the world, from faithfulness, meaning does it match the context you provided.

These axes overlap awkwardly, and recent work says so directly: a model can be factually correct while unfaithful to its evidence, or factually wrong while faithfully reproducing a misleading source.

Framework adapted from the factuality and faithfulness distinction in Huang et al., ACM Transactions on Information Systems (2025).

A worked example makes it stick. Your source says a company was founded in 2001. The model outputs 1998, which happens to be the truth. Factually correct. Unfaithful to the input. Under most definitions that is an intrinsic hallucination, and in a summarisation pipeline it is a defect.

The five failure modes

Those axes are diagnostic rather than operational. Here they are as five modes you can act on.

These categories are not invented here. The AI Hallucination Cases Database maintained by researcher Damien Charlotin tags every entry using labels that map onto the first four: fabricated, misrepresented, false quotes, and outdated advice.

Of the 2,039 court decisions logged as of 12 September 2026, 1,689 involved fabricated material. Another 853 involved a real authority being misrepresented and 549 involved false quotes, with categories overlapping inside a case. Fabrication dominates the headlines. It does not dominate the record by as much as the coverage suggests.

What Is Not a Hallucination

Precision cuts both ways. Several things routinely labelled hallucination are something else, with different remedies:

•      A reasoning error. The model got the facts right and drew a bad conclusion. Fixed by prompting or decomposition, not grounding.

•      Creative output doing its job. When you ask for a short story, invention is the assignment. Context and purpose differ, the mechanism does not.

•      Sycophancy and refusals. Agreeing with a false premise, or refusing a valid request, are distinct behaviours with distinct causes.

•      Classifier false positives and false negatives. Several widely read explainers list these as hallucination examples, citing a fraud model flagging a legitimate transaction. Those are classification errors. No text is generated and nothing is fabricated.

•      Faithfully repeating a false source. If your source was wrong and the model reproduced it accurately, that is a data quality problem.

Why Nobody Agrees on the Hallucination Rate

You have probably seen hallucination rates quoted below 1% and above 40% in the same week. Both can be accurate, because they measure different failure modes on different tasks.

Vectara’s Hughes Hallucination Evaluation Model leaderboard measures summarisation consistency. Models summarise articles, each summary is scored, and any score below 0.5 counts as hallucinated. That is contradiction, the second mode above. PersonQA measures factual recall about people, closer to fabrication. Quoting one against the other is a category mistake. Refusals confound both: tracked separately as an answer rate, a model that declines often scores well while being far less useful.

So are they getting better or worse?

It depends on the task, and saying so matters more than picking a side. Grounded summarisation improved steadily. Open-ended factual recall regressed with the reasoning-model generation, as the chart above shows.

Measurement moved too. An independent comparison of two Vectara snapshots covering 157 models found the mean rate rising from 6.25% to 10.24%, and the share of models under 5% falling from 44% of the field to 9%. Read that before alarm sets in: Vectara refreshed the benchmark in late 2025 with longer documents running up to 32,000 tokens across law, medicine, finance and education. Part of that jump is a harder test, not a worse field.

Longer answers hallucinate more

One finding shows up independently in two places. In the Stanford evaluation below, the tool with the highest hallucination rate also produced the longest answers, averaging 350 words against 219 for its closest competitor. More words means more falsifiable propositions. The same pattern appeared in the 2026 leaderboard snapshot, where summaries of 125 words or more hallucinated at 13.0% against 8.1% for shorter ones.

A longer answer is not a more reliable one. It is a larger surface to check.

Five questions before you believe any rate

1.      Fabrication, contradiction, or something else? The number means nothing without this.

2.      Summarisation, open-ended recall, or a domain task?

3.      Which dataset version? Rates jump when benchmarks are refreshed.

4.      What was the answer rate? Abstention flatters a score.

5.      Who ran it? Vendor evaluations of vendor products are self-assessments.

What This Costs in Practice

Courts produce something unusual: a public, continuously updated record of hallucinations that were caught. Charlotin’s database logs only decisions where a court found or clearly implied that a party relied on hallucinated material. Allegations do not qualify.

Data: AI Hallucination Cases Database, Damien Charlotin (CC BY 4.0), as of 12 September 2026.

As of 12 September 2026 it held 2,039 decisions: 1,396 in the United States, 217 in Canada, 110 in Australia, and 69 in the United Kingdom. Self-represented litigants account for 1,173 and practising lawyers for 811.

Treat that as a floor, not a count. A fabricated citation nobody checks never reaches a docket, and never reaches the database.

Liability sits with the deployer, not the model

The clearest ruling here is Moffatt v. Air Canada, decided by the British Columbia Civil Resolution Tribunal on 14 February 2024. Jake Moffatt asked the airline’s chatbot about bereavement fares before booking flights to a funeral. It said a reduced rate could be claimed retroactively within 90 days. Air Canada’s actual policy did not permit that.

Air Canada argued the chatbot was a separate legal entity responsible for its own actions. The tribunal rejected that, held it was part of Air Canada’s website, found the company owed Moffatt a duty of care, and mostly allowed his claim.

Every organisation putting a model in front of customers inherits that logic. So what actually reduces the risk?

What Actually Reduces Hallucination

Grounding and retrieval, and what they leave untouched

Retrieval-augmented generation supplies source material so the model has something to answer from. It addresses fabrication directly, introduces contradiction risk because now there is a source to contradict, and does little for misattribution. Vendors rarely mention the last two. A preregistered Stanford study, published in the Journal of Empirical Legal Studies in 2025, tested purpose-built legal research tools against that claim.

LexisNexis had advertised hallucination-free linked legal citations. Thomson Reuters said its tools avoided hallucinations by relying on trusted content. The researchers concluded that providers’ claims are overstated. Grounding reduced hallucination. It did not eliminate it.

Calibration and abstention

If the root cause is a scoring system that punishes uncertainty, the fix belongs at the scoring layer. The Kalai paper argues for rewriting the grading on dominant benchmarks so abstention is not penalised. The authors are blunt about the interim: prompts and prompt engineering are of little use against an incentive baked in during training.

Detection by measuring uncertainty about meaning

Farquhar’s method starts from a simple observation. Naive uncertainty measures treat "Paris", "It’s Paris" and "France’s capital Paris" as three answers, when they are one answer phrased three ways. Semantic entropy clusters responses sharing a meaning first, so low entropy means the model is confident about the meaning, not the wording.

It needs no task-specific training data and targets confabulation specifically, which is exactly the narrowing this article has argued for.

What does not work

•      Telling the model not to hallucinate. There is no mode where it stops.

•      Asking it to rate its confidence in words. That rating is generated the same way the answer was.

•      Assuming a more capable model is a more reliable one. The PersonQA numbers settle that.

•      Counting human review as a control without budgeting the time. Checking a 350-word answer properly takes longer than writing one.

Can It Ever Be Eliminated?

There is a formal argument that it cannot. A 2024 paper using computability and diagonalisation arguments concludes that hallucination is an innate limitation of any computable language model, whatever its architecture. That is a statement about theoretical limits, not a prediction that rates cannot fall a long way.

The practical picture is more encouraging, and it comes from researchers who insist hallucination is not mysterious: a known statistical pressure with identified causes, addressable by changing incentives. One obstacle is structural. Some questions have no answer at all, such as an unlisted birthday, so accuracy metrics will always favour guessing at the margin.

Which suggests the goal was wrong. Zero hallucination is the wrong target. Reliable calibration is the right one: a model whose confidence tracks its accuracy closely enough that you know which claims to check. That is achievable, measurable, and more useful than a promise of perfection.

Why the Word You Choose Changes What You Do

Hallucination made a real phenomenon legible to millions who had never thought about token prediction. That is why it stuck. It also smuggled in three assumptions, each leading somewhere costly.

That the model perceives, which invites you to treat it as a mind that occasionally misfires rather than a system doing exactly what it was trained to do. That the failure is exceptional, which invites you to wait for a patch instead of designing verification into the workflow. And that there is one problem, which invites you to buy one solution for five failures and wonder why four survived.

So stop asking whether a model hallucinates. Every one of them does, and the answer tells you nothing you can act on. Ask instead which of the five failure modes your deployment is exposed to, how you would detect that mode, and what your evidence says the rate is on a task resembling yours.

The lawyer in Mata v. Avianca did not fail because ChatGPT hallucinated. He failed because he asked the system that produced the citations whether the citations were real. Knowing which failure you face tells you where verification has to come from. That is the whole practical value of getting the word right.

The Bottom Line

"Hallucination" is one word doing the work of five. Fabrication, contradiction, unsupported extrapolation, misattribution, and stale assertion each come from a different place and each need a different control. Any advice that treats them as one problem is selling you one fix for four failures it will not touch.

Three things follow from that, and they are the ones worth carrying out of this piece:

• A hallucination rate without a definition attached is not a number. Ask which failure mode was measured, on which task, on which dataset version, and what the answer rate was. Most quoted figures survive none of those questions.
• Grounding narrows the problem, it does not close it. The best-performing purpose-built legal tool Stanford tested was still wrong or misgrounded on roughly one query in six, while its marketing promised hallucination-free citations.
• Calibration is the achievable goal, not zero. A model whose confidence tracks its accuracy tells you which claims to check. A model that is confidently right most of the time tells you nothing.

The verification has to come from outside the system that produced the answer. Everything else is detail.