Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Emily M. BenderAlexander Koller
Establishes foundational theoretical limits for large language models by demonstrating that systems trained exclusively on raw text form cannot, in principle, acquire semantic meaning or achieve genuine natural language understanding without grounding in external intent.
Recent advancements in large neural language models have spurred widespread enthusiasm and claims that artificial intelligence systems can now "understand" human language, comprehend text, and recall factual knowledge. However, this hype obscures a fundamental theoretical limitation in how these models operate. The article addresses this critical issue by evaluating whether computational systems trained solely on text prediction can learn the true meaning of natural language expressions.
The main objective of the article is to demonstrate that systems trained exclusively on linguistic form cannot, in principle, acquire meaning. To establish this, the article employs theoretical analysis, conceptual thought experiments, and a review of research in linguistics, developmental psychology, and diagnostic machine learning studies.
The article outlines several foundational findings. First, linguistic meaning is defined as the relation connecting an observable language form to non-linguistic communicative intent and the external world. Second, statistical models trained only on string sequences—exemplified by an isolated octopus learning to mimic human telegraph patterns—can replicate surface lexical correlations but cannot ground words to real-world objects or intentions. Third, empirical diagnostics demonstrate that models that appear to understand complex tasks often rely on statistical artifacts and syntactic heuristics rather than genuine reasoning, causing their performance to collapse when presented with adversarial tests. Fourth, evidence from child language acquisition confirms that humans require interactive joint attention and real-world grounding, rather than passive text exposure, to learn meaning.
These findings have significant implications for technology leadership and AI strategy. Overstating language model capabilities creates severe operational, safety, and reputational risks, particularly if organizations deploy text-only models into environments that demand reliable real-world reasoning. While these models capture useful statistical properties of language structure, they are fundamentally incomplete for human-analogous language understanding.
To build more robust systems, the article recommends pairing linguistic forms with grounding data, such as perceptual inputs or interactive task feedback. Organizations and researchers should also design benchmark tasks that minimize dataset-specific cues and implement rigorous diagnostic evaluations to verify whether systems perform well for valid reasons. While the article acknowledges that language models serve as effective components in broader pipelines and can manipulate surface form exceptionally well, leaders must maintain healthy skepticism regarding claims of genuine machine comprehension.
- Paper: The Symbol Grounding Problem, Stevan Harnad (1990). Introduces the foundational symbol grounding problem, establishing the theoretical argument that symbols cannot acquire real-world meaning purely through relations to other symbols.
- Paper: Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference, R. Thomas McCoy et al. (2019). Demonstrates empirically that language models rely on superficial syntactic heuristics rather than true semantic inference, providing direct evidence for the paper's critique of form-based learning.
- Paper: Adversarial Examples for Evaluating Reading Comprehension Systems, Robin Jia et al. (2017). Exposes how high-performing reading comprehension systems collapse when faced with surface distractors, illustrating that benchmark success does not equate to linguistic understanding.
- Paper: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Jacob Devlin et al. (2019). Introduces the self-supervised masked language modeling paradigm whose benchmark achievements prompted the debate over whether pre-trained models capture true linguistic meaning.
- Paper: On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜, Emily M. Bender et al. (2021). Directly extends the form-versus-meaning argument to large generative models, coining the concept of 'stochastic parrots' to critique the risks of equating scale and fluency with understanding.
- Paper: Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?, Sewon Min et al. (2022). Demonstrates empirically that in-context learning is largely driven by surface task formats rather than ground-truth semantic mappings, substantiating the claim that models primarily manipulate form.
- Paper: Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting, Miles Turpin et al. (2023). Shows that chain-of-thought rationales often function as unfaithful post-hoc justifications rather than true reasoning chains, reinforcing the distinction between surface text generation and genuine comprehension.
- Paper: A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions, Lei Huang et al. (2023). Provides a comprehensive taxonomy of hallucination in large language models, surveying the concrete failures that arise when ungrounded models generate plausible form detached from factual meaning.
- Paper: Unifying Large Language Models and Knowledge Graphs: A Roadmap, Shirui Pan et al. (2023). Outlines architectural pathways to integrate explicit symbolic knowledge graphs with neural language models to mitigate grounding deficiencies.
