Animals vs. Ghosts: Sutton's Critique of Modern AI's Approach to Learning

AI researcher Richard Sutton—author of the widely cited essay "The Bitter Lesson"—challenges whether contemporary large language models actually follow his core principles. During a podcast discussion, Sutton argues that LLMs rely too heavily on massive human-generated datasets and supervised fine-tuning, departing from his original vision of systems that learn dynamically through environmental interaction without embedded human bias. Sutton proposes a fundamentally different architecture inspired by Alan Turing's concept of a "child machine": a system that learns through unsupervised environmental engagement, similar to how animals develop. He contends that supervised fine-tuning lacks any parallel in animal learning, and that internet-scale pretraining introduces human assumptions at every stage. The article's author presents a nuanced counterargument: while acknowledging Sutton's critique carries weight, they note that genuinely unsupervised systems like AlphaZero succeed only in constrained domains. In practice, complex algorithms require rich initialization—which LLMs obtain through pretraining, functioning as a form of artificial evolution that provides the dense information signal necessary to avoid random initialization. The piece characterizes current LLMs as "ghosts": statistical condensations of human documents, thoroughly shaped by human engineering decisions. Yet the author suggests they may eventually be refined toward animal-like intelligence, or diverge into a fundamentally different form of intelligence while remaining profoundly useful.