Hallucinations in Large Language Models: Root Causes, Measurement, and Solutions

Large language models frequently generate fabricated, inconsistent, or unverified information—a phenomenon termed hallucination. This challenge emerges during two critical training phases: initial pretraining on massive internet data (often outdated or incorrect) and subsequent fine-tuning, where introducing new knowledge paradoxically amplifies factual errors. Researchers have developed methods to quantify this problem. Recent studies reveal that when models undergo fine-tuning with new information, they acquire it more slowly than existing knowledge and subsequently exhibit heightened hallucination rates. Evaluation frameworks such as TruthfulQA, FActScore, and SAFE enable assessment of factual accuracy, with findings indicating that larger models do not automatically generate more reliable text. A related concern involves model awareness of knowledge boundaries. When presented with unanswerable or ambiguous questions, models should either refuse to answer or express uncertainty. Research shows that larger models better distinguish answerable from unanswerable questions, yet often display unwarranted confidence in incorrect predictions. Multiple strategies have emerged to enhance factuality, ranging from retrieval-augmented approaches to automated fact verification and response editing. Techniques such as RARR, which retrofits citations to generated text, and FAVA, which combines knowledge retrieval with controlled editing, demonstrate promise, though each method involves tradeoffs between comprehensive coverage and factual precision.