keyword
semantic information
Semantic information is the meaning conveyed by linguistic expressions, distinguishing the conceptual content of words, phrases, and sentences from their surface forms and grammatical structures. In linguistics and natural language processing, it encompasses the representation of entity identities, thematic and semantic roles, word senses, and coreference relationships, capturing how entities relate to actions and events within a broader context. While surface and syntactic levels govern word order, parts of speech, and structural hierarchy, semantic information provides the substantive interpretation necessary to resolve contextual ambiguities and comprehend the intended message of a text.
2 items
What Does BERT Learn about the Structure of Language?
Ganesh Jawahar, Benoît Sagot, Djamé Seddah
Why you should read this
Demonstrates how BERT encodes a bottom-to-top hierarchy of surface, syntactic, and semantic information across its layers while implicitly representing tree-like compositional structures to resolve long-distance dependencies.
BERT is a recent language representation model that has surprisingly performed well in diverse language understanding benchmarks. This result indicates the possibility that BERT networks capture structural information about language. In this work, we provide novel support for this claim by performing a series of experiments to unpack the elements of English language structure learned by BERT. We first show that BERT’s phrasal representation captures phrase-level information in the lower layers. We also show that BERT’s intermediate layers encode a rich hierarchy of linguistic information, with surface features at the bottom, syntactic features in the middle and semantic features at the top. BERT turns out to require deeper layers when long-distance dependency information is required, e.g. to track subject-verb agreement. Finally, we show that BERT representations capture linguistic information in a compositional way that mimics classical, tree-like structures.
Added
2026-09-24
BERT Rediscovers the Classical NLP Pipeline
Ian Tenney, Dipanjan Das, Ellie Pavlick
Why you should read this
Reveals that BERT encodes linguistic features sequentially across its layers in the exact order of the traditional NLP pipeline, from part-of-speech tagging to coreference resolution, while dynamically updating earlier representations using higher-level context.
Pre-trained text encoders have rapidly advanced the state of the art on many NLP tasks. We focus on one such model, BERT, and aim to quantify where linguistic information is captured within the network. We find that the model represents the steps of the traditional NLP pipeline in an interpretable and localizable way, and that the regions responsible for each step appear in the expected sequence: POS tagging, parsing, NER, semantic roles, then coreference. Qualitative analysis reveals that the model can and often does adjust this pipeline dynamically, revising lower-level decisions on the basis of disambiguating information from higher-level representations.
Added
2026-09-17