Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

phrase-level information

Phrase-level information refers to the linguistic properties, structural characteristics, and compositional meanings associated with multi-word syntactic units or constituent spans rather than individual isolated words or entire sentences. In linguistics and natural language processing, this information encompasses grammatical constituent categories, phrasal boundaries, internal modifier relationships, and the emergent semantic meaning produced by combining words within a local span. Computational language models encode phrase-level information within intermediate representations to capture hierarchical sentence structure, enabling systems to identify how words group together into functional syntactic building blocks before resolving larger sentence-level relations and long-distance dependencies.

1 item

What Does BERT Learn about the Structure of Language?

What Does BERT Learn about the Structure of Language?

Ganesh Jawahar, Benoît Sagot, Djamé Seddah

OrganizationsINRIA

Why you should read this

Demonstrates how BERT encodes a bottom-to-top hierarchy of surface, syntactic, and semantic information across its layers while implicitly representing tree-like compositional structures to resolve long-distance dependencies.

BERT is a recent language representation model that has surprisingly performed well in diverse language understanding benchmarks. This result indicates the possibility that BERT networks capture structural information about language. In this work, we provide novel support for this claim by performing a series of experiments to unpack the elements of English language structure learned by BERT. We first show that BERT’s phrasal representation captures phrase-level information in the lower layers. We also show that BERT’s intermediate layers encode a rich hierarchy of linguistic information, with surface features at the bottom, syntactic features in the middle and semantic features at the top. BERT turns out to require deeper layers when long-distance dependency information is required, e.g. to track subject-verb agreement. Finally, we show that BERT representations capture linguistic information in a compositional way that mimics classical, tree-like structures.

Added

2026-09-24