Why AI makes things up, and how to catch it
AI can invent facts, quotes and links, and sound completely sure about them. Here’s why it happens, and five simple habits that help you catch it.
Video transcript
AI assistants can be brilliant. They can also invent a fact, a quote or even a whole book, and sound completely sure about it. Here’s why, and how to catch it.
It might quote someone who never said it, recommend a book that doesn’t exist, give you a link that goes nowhere, get a sum wrong, or tell you something out of date.
These mistakes are called hallucinations. They happen because AI writes by predicting words that sound like a good answer. It doesn’t stop to check whether they’re true.
And it can fool experts. In one study, consultants who used AI on a task it wasn’t good at were 19 percentage points less likely to get the right answer than colleagues without it.
So, five habits. Open the sources yourself. Ask it to flag what it’s unsure of. Check a second source. Use web search for anything recent. And take extra care with health, money and the law.
Not everything needs the same care. Brainstorming names or rewording an email is low risk. Your health, your money, the law, or a quote you plan to repeat? Check carefully.
Use AI for a first answer, never the last word. Read the full guide below for a prompt that makes it show its weak spots.
In 30 seconds
- AI can invent facts, quotes, links and sums, and sounds just as confident when it does.
- It predicts what a good answer sounds like rather than checking whether it’s true.
- Open its sources, check a second one, and take extra care with health, money and the law.
Ask an AI assistant for a good book on the history of your town and it might suggest one with a convincing title, a real-sounding author and a year of publication. Then you try to order it, and find it doesn’t exist.
What a hallucination looks like
When AI states something false as if it were fact, it’s called a Jargon busterHallucination: When AI states something false as if it were true, because it predicts plausible words rather than checking facts.. It isn’t lying, and it isn’t a rare glitch. It’s a side effect of how these tools work, so it can turn up in any answer.
- Made-up facts, quotes and references, including studies, books and court cases that don’t exist.
- Links that go nowhere, or to a page that doesn’t say what the AI claims.
- Wrong sums, especially with several steps or lots of figures.
- Out-of-date information, such as old prices, rules or opening hours.
Why it happens
Assistants like ChatGPT, Claude and Gemini run on a Jargon busterLarge language model: An AI model trained on vast amounts of text to predict the next word. It powers assistants like ChatGPT, Claude and Gemini., which writes by predicting likely words. It produces what a good answer sounds like. Usually that matches the truth. When it doesn’t know, it still produces something that sounds right.
It also likes to please. If your question assumes something false, it may play along rather than correct you. That habit is called Jargon busterSycophancy: When an AI tells you what you want to hear, agreeing with you instead of giving an honest view.. So ask neutral questions: “Is this true?” rather than “Why is this true?”
And its skills are uneven. Researchers call this the Jargon busterJagged frontier: Researchers’ name for AI’s uneven abilities: excellent at some tasks, surprisingly poor at others that look similar.: AI can ace one task and fumble a similar-looking one, with no warning which is which. That catches out experts too.
less likely to reach the right answer: consultants who used AI on a task outside its abilities, compared with those who worked without it.
Source: Harvard Business School, Navigating the Jagged Technological Frontier, September 2023How to catch it
- Ask where it came fromAsk for its sources, then open them yourself. Check each page exists and actually says what the AI claims.
- Ask what it’s unsure ofAsk it to flag anything it isn’t confident about. It won’t catch everything, but it often points you to the weak spots.
- Check a second sourceConfirm any fact you’ll rely on somewhere independent, such as an official website or a trusted news site.
- Search for anything currentFor news, prices, rules or opening hours, ask it to search the web, so it works from today’s pages rather than memory.
- Take extra careWith health, money or the law, use AI to understand the question, then check with a professional or an official source, such as the NHS website or Citizens Advice.
| Lower risk | Check carefully |
|---|---|
| Brainstorming names or gift ideas | Symptoms, medicines and doses |
| Rewording an email you wrote | Money, tax and benefits |
| Summarising something you’ve read | Legal rights and contracts |
| Explaining a topic you can check | Quotes, statistics and references |
Answer my question below. Then list the parts of your answer you’re least sure about, say why, and tell me how I could check each one. My question: [your question]
None of this means you should stop using AI. Treat its answers as a strong first draft, never the final word. For health worries, read how to use AI for health questions, and for better answers from the start, see how to write a good prompt.
Check yourself
3 quick questions nothing is savedTools in this guide
Sources (1)
- Navigating the Jagged Technological FrontierHarvard Business School, September 2023
Spotted a mistake? Tell us and an editor will check it.
