BERT Rediscovers the Classical NLP Pipeline

Ian TenneyDipanjan DasEllie Pavlick

article2019ACL2,046 citations

Reveals that BERT encodes linguistic features sequentially across its layers in the exact order of the traditional NLP pipeline, from part-of-speech tagging to coreference resolution, while dynamically updating earlier representations using higher-level context.

Listen

Modern deep learning language models, such as BERT, have dramatically improved artificial intelligence performance on natural language processing tasks, largely replacing traditional, rule-based computational pipelines. However, these massive neural networks operate as black boxes, making it difficult to understand whether they truly learn meaningful language rules or merely exploit complex surface-level statistics.

The article set out to evaluate where specific types of linguistic information are stored within BERT's internal network layers and to demonstrate how information flows through the model during sentence processing.

The authors applied an edge-probing framework to evaluate frozen 12-layer and 24-layer BERT models across eight benchmark linguistic tasks, spanning low-level syntax to high-level semantics. To evaluate layer-by-layer behavior, the authors introduced two complementary metrics: scalar mixing weights, which reveal which layers the probing models rely on most heavily, and cumulative scoring, which measures the performance gained as successive encoder layers are exposed.

The findings show that BERT organizes linguistic processing in a clear hierarchical order that mirrors the classical language processing pipeline: basic part-of-speech tagging is resolved first, followed by syntactic parsing (constituents and dependencies), named entities, semantic role labeling, and finally coreference. Syntactic information is localized tightly within specific early-to-middle layers, whereas complex semantic tasks require information dispersed across almost all layers. Many simpler linguistic ambiguities are resolved early (layers 1 to 7 in the 24-layer model), while the most informative representations for difficult cases concentrate in later layers (layers 9 to 20). Furthermore, this layer progression scales proportionally between the 12-layer and 24-layer models, exhibiting a consistent structural stretching effect.

Qualitative sentence-level analysis reveals that BERT does not strictly follow a rigid one-way pipeline. Instead, it dynamically revises early, low-level classifications based on context and higher-level semantic understanding discovered in deeper layers. This ability to handle bidirectional feedback explains BERT's superior performance over traditional, sequential processing pipelines that suffer from cascading errors.

Organizations developing or deploying language models should leverage these structural insights to design more efficient architectures, such as selectively extracting intermediate layer representations for lighter-weight tasks rather than running full, compute-heavy networks. Future work should combine structural probing with behavioral evaluations across other modern architectures to further validate how these internal representations drive end-task performance.

These conclusions are based on probing frozen BERT encoders on English benchmark datasets. Because probing classifiers identify the presence of information rather than directly proving how the host model uses it during downstream tasks, readers should treat the results as strong structural evidence of linguistic capability rather than a definitive explanation of all generative decisions.

Brown University Google encoders named entity recognition parsing algorithm BERT Layers Model

arXiv: 1905.05950 nyu-mll/jiant-v1-legacy

Paper: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Jacob Devlin et al. (2019). This paper introduces the core BERT architecture and pre-training objectives whose internal layer-wise representations and linguistic pipeline properties are probed and analyzed by the source.
Paper: Deep contextualized word representations, Matthew E. Peters et al. (2018). This work introduces deep contextualized representations across neural layers, providing the conceptual foundation for probing hierarchical syntactic and semantic features across network depth.
Paper: Natural Language Processing (almost) from Scratch, Ronan Collobert et al. (2011). This seminal paper establishes the unified neural framing of the classical NLP pipeline tasks—POS tagging, chunking, NER, and semantic role labeling—that BERT is shown to internally replicate.
Paper: A unified architecture for natural language processing: deep neural networks with multitask learning, Ronan Collobert et al. (2008). This foundational work introduces multi-task deep neural architectures across classical linguistic tasks, defining the traditional NLP hierarchy explored within BERT's layers.
Paper: Automatic Labeling of Semantic Roles, Daniel Gildea et al. (2000). This pioneering study formalizes semantic role labeling, one of the key intermediate semantic tasks probed in BERT's higher layers.
Paper: Head-Driven Statistical Models for Natural Language Parsing, Michael Collins (2003). This foundational work outlines the syntactic parsing layer of the classical NLP pipeline that BERT is demonstrated to encode in its early-to-middle layers.

Paper: Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning, Armen Aghajanyan et al. (2021). This study analyzes the intrinsic dimensionality of pre-trained language models, extending our understanding of how structured task representations within BERT enable efficient fine-tuning.
Paper: MoEfication: Transformer Feed-forward Layers are Mixtures of Experts, Zhengyan Zhang et al. (2022). This paper examines the modularity and functional specialization of feed-forward layers in Transformers, complementing the layer-wise linguistic pipeline findings of the source.
Paper: SpanBERT: Improving Pre-training by Representing and Predicting Spans, Mandar Joshi et al. (2019). This work adapts BERT to explicitly represent contiguous text spans, advancing its capability on span-centric pipeline tasks like coreference and question answering.
Paper: DeBERTa: Decoding-enhanced BERT with Disentangled Attention, Pengcheng He et al. (2021). This work enhances BERT's representational capacity by disentangling content and position attention, building directly on insights into how contextualized encoders capture syntax and semantics.
Paper: RoBERTa: A Robustly Optimized BERT Pretraining Approach, Yinhan Liu et al. (2019). This paper presents an extensively optimized replication and pre-training recipe for BERT, demonstrating how improved training dynamics solidify the representations studied in the source.
Paper: VisualBERT: A Simple and Performant Baseline for Vision and Language, Liunian Harold Li et al. (2019). This paper extends BERT's multi-layer representation mechanisms to align multimodal visual and linguistic structures in a joint architecture.

Cover for BERT Rediscovers the Classical NLP Pipeline

Abstract

Pre-trained text encoders have rapidly advanced the state of the art on many NLP tasks. We focus on one such model, BERT, and aim to quantify where linguistic information is captured within the network. We find that the model represents the steps of the traditional NLP pipeline in an interpretable and localizable way, and that the regions responsible for each step appear in the expected sequence: POS tagging, parsing, NER, semantic roles, then coreference. Qualitative analysis reveals that the model can and often does adjust this pipeline dynamically, revising lower-level decisions on the basis of disambiguating information from higher-level representations.

Topics

encoders named entity recognition parsing algorithm pos tagging word sense disambiguation syntactic parsing language model

Keywords

BERT Layers Model Layer NLP Pipeline Classical NLP Pipeline BERT Rediscovers Probing Tasks Weights

1 Introduction
2 Model
3 Metrics
3.1 Scalar Mixing Weights
3.2 Cumulative Scoring
4 Results
4.1 Per-Example Analysis
5 Conclusion
References
A Appendix
A.1 Comparison of Encoders
A.2 Additional Examples

Knowls

Knowl 1 — Order of Linguistic Information Representation in BERT Across Model Depth
empirical result
When probing the layer representations of pre-trained BERT across structured linguistic tasks, the progression in which information is extracted closely mirrors the sequence of the traditional natural language processing pipeline. In the 24-layer BERT-large model, as measured by both differential performance gains and scalar mixing weight centers of gravity:
- Basic syntactic tagging is resolved earliest: part-of-speech (POS) tagging is concentrated at the lowest layers (expected layer $\approx 3.39$ , center of gravity $\approx 11.68$ ).
- Syntactic structural relationships follow: syntactic constituents (expected layer $\approx 3.79$ , center of gravity $\approx 13.06$ ) and syntactic dependencies (expected layer $\approx 5.69$ , center of gravity $\approx 13.75$ ).
- Entity labeling is resolved next (expected layer $\approx 4.64$ , center of gravity $\approx 13.16$ ).
- Semantic relationships require higher layers: semantic role labeling (SRL; expected layer $\approx 6.54$ , center of gravity $\approx 13.63$ ).
- Complex semantic phenomena appear latest: coreference resolution (expected layer $\approx 9.47$ , center of gravity $\approx 15.80$ ), semantic proto-roles (SPR; expected layer $\approx 9.93$ , center of gravity $\approx 12.72$ ), and relation classification (SemEval; expected layer $\approx 9.40$ , center of gravity $\approx 12.83$ ).
This sequential hierarchy demonstrates that deep contextualized language models naturally structure hierarchical linguistic representations from shallow syntax to complex semantics across their layers.
Knowl 2 — Layer Pooling via Learned Scalar Mixing Weights and Center of Gravity
model/method
To aggregate representations across layers of an $L$ -layer deep encoder without adding substantial parameter overhead, a task-specific probing classifier uses learned scalar mixing weights. Given encoder token activation vectors $h_i^{(\ell)}$ for token index $i$ at layer $\ell \in \{0, 1, \dots, L\}$ (where layer $0$ denotes non-contextual input embeddings and layer $L$ is the final encoder layer), the pooled contextual representation $h_{i,\tau}$ for task $\tau$ is defined as:

$h_{i,\tau} = \gamma_\tau \sum_{\ell=0}^L s_\tau^{(\ell)} h_i^{(\ell)}$

where $\gamma_\tau \in \mathbb{R}$ is a learned scaling parameter, $a_\tau = (a_\tau^{(0)}, \dots, a_\tau^{(L)}) \in \mathbb{R}^{L+1}$ are learnable layer weights, and $s_\tau = \text{softmax}(a_\tau)$ produces normalized mixing coefficients $s_\tau^{(\ell)} \ge 0$ with $\sum_{\ell=0}^L s_\tau^{(\ell)} = 1$ .

To summarize which layers the probing classifier relies on most heavily, the mixing weight center of gravity $\bar{E}_s[\ell]$ is defined as the expectation of layer depth under the mixing distribution:

$\bar{E}_s[\ell] = \sum_{\ell=0}^L \ell \cdot s_\tau^{(\ell)}$

A higher center-of-gravity value indicates that the probing classifier relies predominantly on higher encoder layers to extract the features relevant to task $\tau$ .
Knowl 3 — Cumulative Scoring, Differential Layer Score, and Expected Layer Metric
model/method
To identify the specific layer at which a pre-trained encoder resolves the linguistic information necessary for a task $\tau$ , cumulative probing classifiers $\{P_\tau^{(\ell)}\}_{\ell=0}^L$ are trained. For each $\ell \in \{0, \dots, L\}$ , classifier $P_\tau^{(\ell)}$ has access only to layers $0, 1, \dots, \ell$ , pooled via scalar mixing. Here, $P_\tau^{(0)}$ represents a non-contextual baseline using only word/subword embeddings, and $P_\tau^{(L)}$ corresponds to probing all layers of the encoder.

The marginal contribution of observing layer $\ell$ (for $\ell \ge 1$ ) is quantified by the differential score $\Delta_\tau^{(\ell)}$ :

$\Delta_\tau^{(\ell)} = \text{Score}(P_\tau^{(\ell)}) - \text{Score}(P_\tau^{(\ell-1)})$

where $\text{Score}(\cdot)$ denotes micro-averaged F1 on an evaluation dataset.

The summary statistic expected layer $\bar{E}_\Delta[\ell]$ is defined as the normalized pseudo-expectation over encoder layer improvements:

$\bar{E}_\Delta[\ell] = \frac{\sum_{\ell=1}^L \ell \cdot \Delta_\tau^{(\ell)}}{\sum_{\ell=1}^L \Delta_\tau^{(\ell)}}$

This measures the average layer at which non-trivial examples (those not already solved at embedding layer $0$ ) are successfully resolved by the contextual encoder.

Knowl 4 — Summary Performance and Layer Localization Statistics Across Edge Probing Tasks

data/table

Probing classifiers evaluated on the 24-layer BERT-large and 12-layer BERT-base models show baseline F1 scores ( $P^{(0)}$ using only non-contextual embeddings), full model F1 scores ( $P^{(L)}$ ), expected layer $\bar{E}_\Delta[\ell]$ , and mixing weight center of gravity $\bar{E}_s[\ell]$ across eight linguistic tasks:

	BERT-base (12 layers)				BERT-large (24 layers)
Task	F1 $P^{(0)}$	F1 $P^{(12)}$	$\bar{E}_\Delta[\ell]$	$\bar{E}_s[\ell]$	F1 $P^{(0)}$	F1 $P^{(24)}$	$\bar{E}_\Delta[\ell]$	$\bar{E}_s[\ell]$
POS	88.4	96.5	1.76	5.59	88.5	96.7	3.39	11.68
Constituents	75.2	87.2	2.38	6.01	73.6	87.0	3.79	13.06
Dependencies	85.2	95.2	2.52	6.55	85.6	95.5	5.69	13.75
Entities	89.9	96.0	2.42	6.86	90.6	96.1	4.64	13.16
SRL	79.5	91.1	2.59	6.60	81.3	91.4	6.54	13.63
Coreference	80.0	90.2	3.62	7.96	80.5	91.9	9.47	15.80
SPR	77.8	83.5	4.29	6.59	77.7	83.7	9.93	12.72
Relations	58.9	82.9	4.16	6.80	60.7	84.2	9.40	12.83

The data illustrates two primary patterns:

In both model scales, low-level syntactic tasks (POS, constituents) have lower expected layer and center-of-gravity values than high-level semantic tasks (coreference, SPR, relations).
Comparing BERT-base and BERT-large shows that layer statistics roughly double in BERT-large, demonstrating relative depth preservation across model capacities.

Knowl 5 — Syntactic Localizability Versus Semantic Dispersion Across BERT Layers
empirical result
Probing weight distributions across BERT layers show distinct localization properties depending on whether a task is syntactic or semantic:
- Syntactic tasks (POS tagging, constituent parsing, dependency parsing) exhibit high localization. Their mixing weights $s_\tau^{(\ell)}$ and differential scores $\Delta_\tau^{(\ell)}$ concentrate heavily on a small subset of layers, yielding high Kullback-Leibler divergence from a uniform distribution ( $K(s) = \text{KL}(s_\tau \parallel \text{Uniform})$ and $K(\Delta) = \text{KL}(\Delta_\tau \parallel \text{Uniform})$ ). On BERT-large, dependency parsing achieves $K(s) = 0.87$ and $K(\Delta) = 1.15$ , while constituent parsing achieves $K(s) = 0.83$ and $K(\Delta) = 1.57$ .
- Semantic tasks (semantic relations and semantic proto-roles / SPR) exhibit broad dispersion. Their mixing weights are nearly uniform across layers ( $K(s) = 0.01$ for both SPR and Relations on BERT-large), and differential score improvements are distributed gradually across almost all layers ( $K(\Delta) = 0.33$ for SPR, $0.50$ for Relations).
- Entity labeling exhibits intermediate behavior: many instances are resolved in early layers via surface statistics ( $K(\Delta) = 1.61$ ), but mixing weights show low concentration ( $K(s) = 0.06$ ) with continued refinements in high layers for fine-grained distinctions (e.g., Organization vs. Geopolitical Entity).
Knowl 6 — Discrepancy Between Early Differential Score Improvements and Late Mixing Weight Concentrations
empirical result
When comparing cumulative differential score gains ( $\Delta_\tau^{(\ell)}$ ) and scalar mixing weights ( $s_\tau^{(\ell)}$ ) in BERT-large:
- Differential scores peak early in the encoder (layers 1–7), indicating that a majority of test examples can be classified correctly with shallow contextual information or heuristic shortcuts.
- Scalar mixing weights peak significantly later in the encoder (layers 9–20).
For tasks with concentrated weights, the highest mixing weights appear on or immediately following the highest layers that yield positive incremental improvements ( $\Delta_\tau^{(\ell)} > 0$ ) in validation F1. This occurs because the global probing classifier preferentially attends to the deeper, fully contextualized representations where the most challenging examples are resolved, even if simpler examples could have been solved with fewer layers.
Knowl 7 — Cross-Architecture Representation Comparison: BERT-Base, BERT-Large, and ELMo
empirical result
Comparing layer-wise representations across BERT-base (12 layers), BERT-large (24 layers), and ELMo (2 LSTM layers over a character CNN):
- Stretching Effect between BERT-base and BERT-large: The relative depth at which tasks are encoded is conserved across model sizes. Tasks peak at roughly the same normalized depth relative to the total depth of the encoder, with task order (POS $\rightarrow$ constituents $\rightarrow$ dependencies $\rightarrow$ SRL $\rightarrow$ coreference) preserved across both 12-layer and 24-layer configurations.
- ELMo Layer Usage: In ELMo, the first LSTM layer (layer 1) receives the highest mixing weight across all eight probing tasks, showing limited layer specialization compared to BERT.
- Coreference Representation Differences: While BERT assigns concentrated mixing weights to higher layers for coreference resolution ( $K(s) = 0.50$ in BERT-large, peaking around layers 14–20), ELMo exhibits only weak concentration ( $K(s) = 0.10$ ), indicating that ELMo provides substantially weaker specialized representations for coreference.
Knowl 8 — Dynamic Out-of-Order Linguistic Processing in Individual Sentence Traces
empirical result
Although BERT encodes linguistic abstractions in the sequence of the traditional NLP pipeline in the aggregate, layer-by-layer tracking of per-example predictions reveals that the network executes dynamic, non-sequential adjustments on ambiguous sentences:
- Semantic Feedback Revising Entity Typing: In the sentence "he smoked toronto in the playoffs...", the model initially assigns the token "toronto" the entity label Geopolitical Entity (GPE) in lower layers. Once higher layers resolve the semantic role of "toronto" as the patient (ARG1) of "smoked", the entity label is dynamically revised to Organization (ORG, representing the sports team).
- Higher-Level Semantic Disambiguation of Part-of-Speech and Entity Labels: In the ambiguous phrase "china today blacked out...", early layers classify "today" as a common noun (NN), date entity (DATE), and temporal modifier (ARGM-TMP). In subsequent layers, the network revises "china today" to a proper noun mention (NNP) and Organization (ORG), subsequently updating the semantic role to agent (ARG0).
This demonstrates that deep Transformer representations support bidirectional interactions across linguistic levels, deferring ambiguous decisions or revising low-level hypotheses based on high-level semantic context.
Knowl 9 — Edge Probing Framework on Frozen BERT Representations
experimental setup
The structural representation of linguistic properties in BERT is evaluated using the edge probing protocol of Tenney et al. (2019):
- Input and Targets: Probing classifiers take one or two labeled token spans $s_1 = [i_1, j_1)$ and $s_2 = [i_2, j_2)$ within a tokenized sentence $T = [t_0, \dots, t_n]$ and predict a discrete linguistic label (e.g., constituent type, relation label, semantic role). The probing classifier is restricted to the pooled per-token contextual vectors within the target spans.
- Frozen Encoder: Pre-trained BERT-base and BERT-large uncased models (pre-trained on masked language modeling and next sentence prediction on a 3.3B word English corpus) are evaluated with all Transformer encoder weights held frozen. This prevents the encoder representations from being altered to fit the probing task.
- Probing Tasks: Eight tasks evaluated via micro-averaged F1:
  1. Part-of-speech (POS; OntoNotes 5.0)
  2. Syntactic constituents (OntoNotes 5.0)
  3. Dependency parsing (English Web Treebank)
  4. Named entities (OntoNotes 5.0)
  5. Semantic role labeling (SRL; OntoNotes 5.0)
  6. Coreference resolution (OntoNotes 5.0)
  7. Semantic proto-roles (SPR1 dataset)
  8. Relation classification (SemEval 2010 Task 8).
Knowl 10 — Limitations of Structural Inspection-Based Probing in Deep Language Models
limitation
The methodology of using diagnostic probing classifiers on frozen internal layer representations has two primary limitations:
1. Absence of Proof: The failure of a probing classifier to detect a specific linguistic property in a layer does not prove that the information is absent from that layer; the representations may encode the information in a non-linear or inaccessible format not extracted by the probe architecture.
2. Lack of Functional Causality: The presence of an extractable linguistic pattern within a layer does not demonstrate that the pre-trained model actively uses that information during downstream task inference.
Consequently, structural inspection of layer activations provides an exploratory analysis of representational capacity and must be combined with behavioral and perturbation studies to determine causal utility.

Coverage note — None was omitted; all primary methodological contributions, empirical findings, summary data, cross-model comparisons, per-example dynamic analyses, and stated limitations are fully covered.

References

1.Yonatan Belinkov. 2018. On internal language representations in deep learning: An analysis of machine translation and speech recognition. Ph.D. thesis, Massachusetts Institute of Technology.
2.Terra Blevins, Omer Levy, and Luke Zettlemoyer. 2018. Deep RNNs encode soft hierarchical syntax. In Proceedings of ACL.
3.Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018. What you can cram into a single $&#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of ACL.
4.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL.
5.Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2009. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions.
6.Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multi-task deep neural networks for natural language understanding. arXiv preprint 1901.11504.
7.Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of ACL: System Demonstrations.
8.Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of EMNLP.
9.Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS.
10.Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In Proceedings of NAACL.
11.Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018b. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of EMNLP.
12.Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018. Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of EMNLP.
13.Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. https://blog.openai.com/language-unsupervised.
14.Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. https://blog.openai.com/better-language-models.
15.Drew Reisinger, Rachel Rudinger, Francis Ferraro, Craig Harman, Kyle Rawlins, and Benjamin Van Durme. 2015. Semantic proto-roles. Transactions of the Association of Computational Linguistics.
16.Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural MT learn source syntax? In Proceedings of EMNLP.
17.Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Christopher D. Manning. 2014. A gold standard dependency corpus for English. In Proceedings of the Ninth International Conference on Language Resources and Evaluation.
18.Adam Teichert, Adam Poliak, Benjamin Van Durme, and Matthew Gormley. 2017. Semantic proto-role labeling. In Proceedings of AAAI.
19.Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations.
20.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS.
21.Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. OntoNotes release 5.0 LDC2013T19. Linguistic Data Consortium, Philadelphia, PA.

Citation

MLA

Tenney, I., et al. “BERT Rediscovers the Classical NLP Pipeline”. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 4593–601, https://doi.org/10.18653/v1/P19-1452.

APA

Tenney, I., Das, D., & Pavlick, E. (2019). BERT Rediscovers the Classical NLP Pipeline. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 4593–4601. https://doi.org/10.18653/v1/P19-1452

Chicago

Tenney, I., D. Das, and E. Pavlick. 2019. “BERT Rediscovers the Classical NLP Pipeline”. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 4593–4601. https://doi.org/10.18653/v1/P19-1452.

Harvard

Tenney, I., Das, D. and Pavlick, E. (2019) “BERT Rediscovers the Classical NLP Pipeline”, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pp. 4593–4601. Available at: https://doi.org/10.18653/v1/P19-1452.

Vancouver

1. Tenney I, Das D, Pavlick E (2019) BERT Rediscovers the Classical NLP Pipeline. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pp 4593–4601

BibTeX

@inproceedings{tenney-etal-2019-bert,
    title = "{BERT} Rediscovers the Classical {NLP} Pipeline",
    author = "Tenney, Ian  and
      Das, Dipanjan  and
      Pavlick, Ellie",
    editor = "Korhonen, Anna  and
      Traum, David  and
      M{\`a}rquez, Llu{\'i}s",
    booktitle = "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics",
    month = jul,
    year = "2019",
    address = "Florence, Italy",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/P19-1452/",
    doi = "10.18653/v1/P19-1452",
    pages = "4593--4601"
}

Metadata:ACL Anthology

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

BERT Rediscovers the Classical NLP Pipeline

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Order of Linguistic Information Representation in BERT Across Model Depth

Knowl 2 — Layer Pooling via Learned Scalar Mixing Weights and Center of Gravity

Knowl 3 — Cumulative Scoring, Differential Layer Score, and Expected Layer Metric

Knowl 4 — Summary Performance and Layer Localization Statistics Across Edge Probing Tasks

Knowl 5 — Syntactic Localizability Versus Semantic Dispersion Across BERT Layers

Knowl 6 — Discrepancy Between Early Differential Score Improvements and Late Mixing Weight Concentrations

Knowl 7 — Cross-Architecture Representation Comparison: BERT-Base, BERT-Large, and ELMo

Knowl 8 — Dynamic Out-of-Order Linguistic Processing in Individual Sentence Traces

Knowl 9 — Edge Probing Framework on Frozen BERT Representations

Knowl 10 — Limitations of Structural Inspection-Based Probing in Deep Language Models

References

Citation

BibTeX

Source Code

Access the Paper