keyword

hierarchical syntax

Hierarchical syntax is the organizational framework in linguistics and computational language modeling in which sentences are structured into nested, multi-level constituents rather than simple linear sequences of words. Under this model, individual words combine into phrases and clauses according to grammatical rules, forming a structural arrangement typically represented as a parse tree or dependency hierarchy. Within these tree representations, the relationships between words are defined by structural depth, head-dependent configurations, and syntactic distances rather than their immediate surface proximity. This structural arrangement governs core grammatical phenomena, including constituent boundaries, agreement, and overall semantic interpretation.

1 item

A Structural Probe for Finding Syntax in Word Representations

John Hewitt, Christopher D. Manning

OrganizationsStanford University

Why you should read this

Proposes a geometric probing framework to demonstrate that contextual language models like BERT and ELMo implicitly embed entire hierarchical syntax trees directly within linear subspaces of their vector representations.

Recent work has improved our ability to detect linguistic knowledge in word representations. However, current methods for detecting syntactic knowledge do not test whether syntax trees are represented in their entirety. In this work, we propose a structural probe, which evaluates whether syntax trees are embedded in a linear transformation of a neural network's word representation space. The probe identifies a linear transformation under which squared L2 distance encodes the distance between words in the parse tree, and one in which squared L2 norm encodes depth in the parse tree. Using our probe, we show that such transformations exist for both ELMo and BERT but not in baselines, providing evidence that entire syntax trees are embedded implicitly in deep models' vector geometry.

Source

https://aclanthology.org/N19-1419.pdf

Added

2026-09-25