Why do AI chatbots confidently make up facts and court cases?

An AI chatbot can state a completely false fact with total confidence because of how it was trained. Models like ChatGPT are trained during pre-training to predict the next word in a sentence, and that training incentivizes them to 'give a guess' about what comes next even when they genuinely lack the information. Incomplete or unrepresentative training data makes this worse.

The word 'hallucination' itself has a strange history. It originally meant something useful: since the 2000s, computer-vision researchers used it for adding helpful detail to a blurry image, a term that traces back to a 1986 PhD thesis. By 2015, the meaning shifted when researcher Andrej Karpathy used 'hallucinated' to describe a language model inventing a fake citation link. In 2021, Meta warned its BlenderBot 2 chatbot was prone to 'confident statements that are not true,' and after ChatGPT's late-2022 launch, the term exploded into everyday use; by 2023 the Cambridge dictionary had updated its own definition to include this AI-specific meaning.

The consequences can be more than embarrassing. Data scientist Teresa Kubacka once invented a fake scientific term, 'cycloidal inverted electromagnon,' just to test ChatGPT — and the chatbot confidently invented a plausible-sounding explanation backed by equally fake citations. In May 2023, a lawyer filed six court case citations generated by ChatGPT into a real lawsuit; the cases didn't exist, and the chatbot kept insisting they were real even after their nonexistence was discovered. By April 2026, a database tracking such incidents had logged more than 1,300 cases of AI hallucinations showing up in legal decisions alone.

Recent research suggests part of the mechanism: in 2025, Anthropic's interpretability work on its own Claude model found internal circuits that, by default, make the model decline to answer unless it recognizes it has enough information. Hallucinations, the researchers found, happen when that inhibition misfires, such as when the model recognizes a name but doesn't actually know enough about the person, and generates a plausible-sounding but untrue answer.