keyword
syntactic structures
Syntactic structures are the rule-governed, hierarchical patterns and relationships that organize words into phrases, clauses, and sentences according to the grammar of a natural language. In linguistics and computational language processing, these structures describe how individual lexical units combine compositionally into larger grammatical units, typically modeled as constituent trees or dependency graphs. They specify formal grammatical relations such as subject-verb agreement, phrase nesting, and long-distance dependencies, operating distinctly from linear word sequences and semantic interpretations. By organizing elements into predictable configurations, syntactic structures provide the structural framework necessary to determine grammatical validity, resolve structural ambiguities, and systematically interpret and generate complex expressions.
2 items
The Impact of Depth on Compositional Generalization in Transformer Language Models
Jackson Petty, Sjoerd van Steenkiste, Ishita Dasgupta, Fei Sha, Dan Garrette, Tal Linzen
Why you should read this
Demonstrates that deeper transformer models improve compositional generalization over wider models of equal parameter size but yield rapidly diminishing returns, proving that practitioners can adopt shallower architectures to lower latency without sacrificing performance.
To process novel sentences, language models (LMs) must generalize compositionally -- combine familiar elements in new ways. What aspects of a model's structure promote compositional generalization? Focusing on transformers, we test the hypothesis, motivated by theoretical and empirical work, that deeper transformers generalize more compositionally. Simply adding layers increases the total number of parameters; to address this confound between depth and size, we construct three classes of models which trade off depth for width such that the total number of parameters is kept constant (41M, 134M and 374M parameters). We pretrain all models as LMs and fine-tune them on tasks that test for compositional generalization. We report three main conclusions: (1) after fine-tuning, deeper models generalize more compositionally than shallower models do, but the benefit of additional layers diminishes rapidly; (2) within each family, deeper models show better language modeling performance, but returns are similarly diminishing; (3) the benefits of depth for compositional generalization cannot be attributed solely to better performance on language modeling. Because model latency is approximately linear in the number of layers, these results lead us to the recommendation that, with a given total parameter budget, transformers can be made shallower than is typical without sacrificing performance.
Added
2026-10-04
What Does BERT Learn about the Structure of Language?
Ganesh Jawahar, Benoît Sagot, Djamé Seddah
Why you should read this
Demonstrates how BERT encodes a bottom-to-top hierarchy of surface, syntactic, and semantic information across its layers while implicitly representing tree-like compositional structures to resolve long-distance dependencies.
BERT is a recent language representation model that has surprisingly performed well in diverse language understanding benchmarks. This result indicates the possibility that BERT networks capture structural information about language. In this work, we provide novel support for this claim by performing a series of experiments to unpack the elements of English language structure learned by BERT. We first show that BERT’s phrasal representation captures phrase-level information in the lower layers. We also show that BERT’s intermediate layers encode a rich hierarchy of linguistic information, with surface features at the bottom, syntactic features in the middle and semantic features at the top. BERT turns out to require deeper layers when long-distance dependency information is required, e.g. to track subject-verb agreement. Finally, we show that BERT representations capture linguistic information in a compositional way that mimics classical, tree-like structures.
Added
2026-09-24