Back

How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings

Kawin Ethayarajh

article2019EMNLP1,352 citations

Reveals the geometric behavior of BERT, ELMo, and GPT-2 embeddings across layers, proving that upper layers generate more context-specific representations while static word vectors account for less than five percent of their variance.

Listen

Contextualized language models have dramatically improved modern language processing by generating word meanings based on surrounding text rather than relying on a single static representation. Despite their success, the internal geometry of these models remains poorly understood, leaving unanswered questions about how contextual these representations actually are and whether simpler static vectors could adequately replace them.

The article evaluates the geometry of word representation spaces across all layers of three leading neural models—ELMo, BERT, and GPT-2—to determine the degree of contextuality and assess whether static embeddings can replace contextual representations.

The analysis uses sentences from standard Semantic Textual Similarity datasets (2012–2016) to track words appearing in multiple distinct contexts. The evaluation measures how representations vary across contexts (self-similarity), how words relate to the overall sentence (intra-sentence similarity), and how much variation a single static vector can capture (maximum explainable variance), while adjusting for directional non-uniformity across the vector space.

The findings reveal four key insights. First, representations across all three models are highly anisotropic, meaning word vectors occupy a narrow directional cone rather than spreading uniformly across the vector space, with upper layers showing the most extreme clustering. Second, representations become increasingly context-specific in higher network layers; common words like stopwords exhibit the highest context-specificity due to appearing in diverse contexts. Third, models handle sentence context differently: words in the same sentence become more similar in ELMo's upper layers, become more distinct in BERT, and show virtually no added similarity over random words in GPT-2. Fourth, even in the best-case scenario, a static embedding explains less than 5% of the variance in a word's contextualized representations, proving that models do not simply select from a finite set of dictionary word senses.

These results provide a clear mathematical justification for why contextualized models consistently outperform older static techniques across language tasks. Although large neural models demand substantial memory, computational time, and hosting costs, replacing them entirely with standard static vectors causes significant performance loss. However, extracting static vectors from the principal components of lower layers in models like BERT outperforms traditional GloVe and FastText embeddings on standard benchmarks, providing a viable low-cost deployment option.

Organizations seeking lighter computational footprints should consider extracting static vectors from lower layers of transformer models when production environments cannot host full neural networks. In addition, model developers should explore adding penalties during model training to reduce directional narrowing and potentially boost accuracy. Further research is required to determine why different model architectures process sentence-level context in fundamentally distinct ways.

While the analysis provides robust conclusions based on diverse benchmark tasks, findings are constrained by the sentence datasets examined and focus specifically on base-level models. Practitioners should test these representations against their specific domain data when applying static extractions in production.

Stanford Universitygeometrypart of speechword embeddingcontextualized word representationsGPT-2 embeddingscontextuality measures
arXiv: 1909.00512
Cover for How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings

Abstract

Replacing static word embeddings with contextualized word representations has yielded significant improvements on many NLP tasks. However, just how contextual are the contextualized representations produced by models such as ELMo and BERT? Are there infinitely many context-specific representations for each word, or are words essentially assigned one of a finite number of word-sense representations? For one, we find that the contextualized representations of all words are not isotropic in any layer of the contextualizing model. While representations of the same word in different contexts still have a greater cosine similarity than those of two different words, this self-similarity is much lower in upper layers. This suggests that upper layers of contextualizing models produce more context-specific representations, much like how upper layers of LSTMs produce more task-specific representations. In all layers of ELMo, BERT, and GPT-2, on average, less than 5% of the variance in a word's contextualized representations can be explained by a static embedding for that word, providing some justification for the success of contextualized representations.

Topics

geometrypart of speechword embeddingword sensevisual representationslayeredapplication layerstransport layerslanguage model

Keywords

contextualized word representationsGPT-2 embeddingscontextuality measuresaverage intra-sentence similarityanisotropy baselineneural language modelsstatic embeddingsrepresentation space

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 Approach
  • 3.1 Contextualizing Models
  • 3.2 Data
  • 3.3 Measures of Contextuality
  • 3.4 Adjusting for Anisotropy
  • 4 Findings
  • 4.1 (An)Isotropy
  • 4.2 Context-Specificity
  • 4.3 Static vs. Contextualized
  • 5 Future Work
  • 6 Conclusion
  • References

Knowls

  1. Knowl 1 — Upper Bound on Word Representation Variance Explained by Static Embeddings

    empirical result

    Across all layers of ELMo, BERT (base, cased), and GPT-2, on average less than 5% of the variance in a word's contextualized representations across contexts can be explained by a single static embedding vector, after adjusting for ambient anisotropy.

    Even without adjusting for anisotropy, the raw Maximum Explainable Variance (MEV) averages below 5% for all layers of ELMo and BERT. In GPT-2, raw MEV is non-negligible (averaging ~30% for layers 2 through 11) purely due to severe anisotropy; once anisotropy is subtracted, adjusted MEV drops below 5%. Furthermore, for many individual words, the raw proportion of variance explained by their first principal component is lower than that of a set of randomly sampled word representations. This demonstrates that contextualized language models do not merely assign tokens to a finite set of discrete word senses, and that static embeddings are fundamentally incapable of capturing the variance of contextualized representations.

  2. Knowl 2 — Maximum Explainable Variance (MEV) for Contextualized Representations

    definition

    Let ww be a word that occurs across nn distinct sentence contexts {s1,…,sn}\{s_1, \dots, s_n\} at token indices {i1,…,in}\{i_1, \dots, i_n\} respectively (with n≥5n \ge 5). Let fℓ(s,i)∈Rdf_\ell(s, i) \in \mathbb{R}^d denote the representation of token s[i]s[i] at layer ℓ\ell of a model ff. The occurrence matrix of ww in layer ℓ\ell is the matrix [fℓ(s1,i1),…,fℓ(sn,in)]∈Rd×n[f_\ell(s_1, i_1), \dots, f_\ell(s_n, i_n)] \in \mathbb{R}^{d \times n}. If σ1,…,σm\sigma_1, \dots, \sigma_m (where m=min⁡(d,n)m = \min(d, n)) are the singular values of this matrix sorted in descending order, the Maximum Explainable Variance is defined as:

    MEVℓ(w)=σ12∑i=1mσi2\text{MEV}_\ell(w) = \frac{\sigma_1^2}{\sum_{i=1}^m \sigma_i^2}

    MEVℓ(w)∈[0,1]\text{MEV}_\ell(w) \in [0, 1] represents the proportion of variance across the contextualized representations of ww that can be captured by their first principal component, serving as a theoretical upper bound on how well any static embedding could replace contextualized representations for that word.

  3. Knowl 3 — Self-Similarity Metric and Layerwise Context-Specificity

    definition

    Let ww be a word appearing in sentence contexts {s1,…,sn}\{s_1, \dots, s_n\} at indices {i1,…,in}\{i_1, \dots, i_n\} (n≥5n \ge 5). Let fℓ(s,i)f_\ell(s, i) be the representation of s[i]s[i] at layer ℓ\ell of model ff. The self-similarity of ww at layer ℓ\ell is the mean pairwise cosine similarity between its representations across all distinct contexts:

    SelfSimℓ(w)=1n2−n∑j=1n∑k≠jcos⁡(fℓ(sj,ij),fℓ(sk,ik))\text{SelfSim}_\ell(w) = \frac{1}{n^2 - n} \sum_{j=1}^n \sum_{k \neq j} \cos\left(f_\ell(s_j, i_j), f_\ell(s_k, i_k)\right)

    To account for directionality biases, the anisotropy-adjusted self-similarity is defined by subtracting the baseline expected cosine similarity between uniformly randomly sampled token representations from the set of all word occurrences O\mathcal{O}:

    SelfSimℓ∗(w)=SelfSimℓ(w)−Ex,y∼U(O)[cos⁡(fℓ(x),fℓ(y))]\text{SelfSim}^*_\ell(w) = \text{SelfSim}_\ell(w) - \mathbb{E}_{x, y \sim \mathcal{U}(\mathcal{O})}\left[\cos\left(f_\ell(x), f_\ell(y)\right)\right]

    In ELMo, BERT, and GPT-2, SelfSimℓ∗(w)\text{SelfSim}^*_\ell(w) decreases monotonically as layer depth ℓ\ell increases (e.g., from 1.0 at layer 0 down to ~0.5 at layer 2 in ELMo; from ~0.85 at layer 0 down to ~0.3 at layer 12 in BERT), showing that higher layers produce systematically more context-specific representations. GPT-2 exhibits the lowest self-similarity across all hidden layers.

  4. Knowl 4 — Anisotropy of Contextualized Word Representations

    empirical result

    In all non-input layers of ELMo, BERT, and GPT-2, contextualized word representations are anisotropic: instead of being uniformly distributed across directions in the embedding space (where expected cosine similarity between randomly sampled words is 0), they occupy a narrow cone in vector space.

    Anisotropy generally intensifies in deeper layers:

    • In GPT-2, the average cosine similarity between randomly sampled words is ~0.6 in layers 2–8 and escalates in layers 8–12, approaching almost 1.0 in layer 12 (two randomly sampled words have nearly identical directional vectors).
    • In BERT, random word cosine similarity increases from ~0.1 at layer 0 to ~0.67 at layer 11, before dipping to ~0.45 at layer 12.
    • In ELMo, hidden layers have an average random word cosine similarity of ~0.15 to ~0.22, whereas its non-contextualized character-level input layer (layer 0) is essentially isotropic with average cosine similarity close to 0.

  5. Knowl 5 — Intra-Sentence Similarity and Cross-Architecture Contextualization Geometry

    empirical result

    The intra-sentence similarity of a sentence s=⟨w1,…,wn⟩s = \langle w_1, \dots, w_n \rangle at layer ℓ\ell of model ff is defined as the average cosine similarity between each word vector and the sentence mean vector sˉℓ=1n∑i=1nfℓ(s,i)\bar{s}_\ell = \frac{1}{n} \sum_{i=1}^n f_\ell(s, i):

    IntraSimℓ(s)=1n∑i=1ncos⁡(sˉℓ,fℓ(s,i))\text{IntraSim}_\ell(s) = \frac{1}{n} \sum_{i=1}^n \cos\left(\bar{s}_\ell, f_\ell(s, i)\right)

    After adjusting by subtracting the random-pair anisotropy baseline, intra-sentence similarity reveals fundamentally different geometric behaviors across model families:

    • ELMo: Anisotropy-adjusted intra-sentence similarity increases in upper layers (from ~0.30 at layer 0 to ~0.48 at layer 2), meaning words in the same sentence converge toward one another in vector space as context-specificity increases.
    • BERT: Anisotropy-adjusted intra-sentence similarity decreases in upper layers (from ~0.35 at layer 0 to ~0.20 at layer 11) while remaining strictly positive (> 0.20 across all layers). Words in the same sentence drift apart from each other while remaining more similar than randomly paired words.
    • GPT-2: Anisotropy-adjusted intra-sentence similarity is close to 0 in all hidden layers (and highest at layer 0 at ~0.20). Words in the same sentence are geometrically no more similar to one another than words drawn at random from separate sentences.

  6. Knowl 6 — Extreme Context-Specificity of Stopwords

    empirical result

    Across all layers of ELMo, BERT, and GPT-2, high-frequency functional stopwords (such as 'and', 'of', ''s', 'the', and 'to') exhibit the lowest self-similarity among all words in the vocabulary, meaning their contextualized representations are among the most context-specific.

    Because stopwords possess very few distinct semantic word senses, their high variability across contexts shows that variance in contextualized embeddings is driven by the diversity of syntactic and sentential contexts in which a word appears, rather than the intrinsic polysemy of the word.

  7. Knowl 7 — Performance of Static Word Embeddings Derived from First Principal Components

    data/table

    Static word embeddings can be created by extracting the first principal component (PC) of a word's contextualized representations across contexts in a given model layer. The static embeddings extracted from lower layers of BERT and ELMo outperform standard static word vectors (GloVe and FastText) on word similarity, analogy solving, and concept categorization benchmarks.

    Static Embedding SimLex999 MEN WS353 RW Google MSR SemEval2012(2) BLESS AP
    GloVe 0.194 0.216 0.339 0.127 0.189 0.312 0.097 0.390 0.308
    FastText 0.239 0.239 0.432 0.176 0.203 0.289 0.104 0.375 0.291
    ELMo, Layer 1 0.276 0.167 0.317 0.148 0.170 0.326 0.114 0.410 0.308
    ELMo, Layer 2 0.215 0.151 0.272 0.133 0.130 0.268 0.132 0.395 0.318
    BERT, Layer 1 0.315 0.200 0.394 0.208 0.236 0.389 0.166 0.365 0.321
    BERT, Layer 2 0.320 0.166 0.383 0.188 0.230 0.385 0.149 0.365 0.321
    BERT, Layer 11 0.221 0.076 0.319 0.135 0.175 0.290 0.149 0.370 0.289
    BERT, Layer 12 0.233 0.082 0.325 0.144 0.184 0.307 0.144 0.360 0.294
    GPT-2, Layer 1 0.174 0.012 0.176 0.183 0.052 0.081 0.033 0.220 0.184
    GPT-2, Layer 2 0.135 0.036 0.171 0.180 0.045 0.062 0.021 0.245 0.184
    GPT-2, Layer 11 0.126 0.034 0.165 0.182 0.031 0.038 0.045 0.270 0.189
    GPT-2, Layer 12 0.140 -0.009 0.113 0.163 0.020 0.021 0.014 0.225 0.172

    Static embeddings derived from BERT Layer 1 and Layer 2 achieve the best overall results across most tasks. Across all models, first principal components extracted from lower layers outperform those from higher layers, and GPT-2 principal components perform markedly worse across benchmarks due to high anisotropy and excessive context-specificity.

Coverage note — All substantial contributions—including the three contextuality metrics (self-similarity, intra-sentence similarity, MEV), the anisotropy adjustment formulation, layerwise anisotropy and context-specificity findings across BERT, ELMo, and GPT-2, the stopword finding, and the static benchmark evaluation of PCA embeddings—are fully represented. Intermediate layers 3–10 omitted from Table 1 in the paper are omitted here as well.

References

  1. 1.Eneko Agirre, Carmen Banea, Claire Cardie, Daniel M Cer, Mona T Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, et al. 2015. Semeval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability. In Proceedings SemEval@ NAACL-HLT. pages 252–263.
  2. 2.Eneko Agirre, Carmen Banea, Claire Cardie, Daniel M Cer, Mona T Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. Semeval-2014 task 10: Multilingual semantic textual similarity. In Proceedings SemEval@ COLING. pages 81–91.
  3. 3.Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. Sem 2013 shared task: Semantic textual similarity, including a pilot on typed-similarity. In SEM 2013: The Second Joint Conference on Lexical and Computational Semantics. Association for Computational Linguistics.
  4. 4.Eneko Agirre, Mona Diab, Daniel Cer, and Aitor Gonzalez-Agirre. 2012. Semeval-2012 task 6: A pilot on semantic textual similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation. Association for Computational Linguistics, pages 385–393.
  5. 5.Abdulrahman Almuhareb and Massimo Poesio. 2004. Attribute-based and value-based clustering: An evaluation. In Proceedings of the 2004 conference on empirical methods in natural language processing.
  6. 6.Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In International Conference on Learning Representations.
  7. 7.Marco Baroni and Alessandro Lenci. 2011. How we blessed distributional semantic evaluation. In Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics. Association for Computational Linguistics, pages 1–10.
  8. 8.Elia Bruni, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. Journal of Artificial Intelligence Research 49:1–47.
  9. 9.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 .
  10. 10.Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2002. Placing search in context: The concept revisited. ACM Transactions on information systems 20(1):116–131.
  11. 11.John R Firth. 1957. A synopsis of linguistic theory, 1930-1955. Studies in linguistic analysis .
  12. 12.John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics.
  13. 13.Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics 41(4):665–695.
  14. 14.David A Jurgens, Peter D Turney, Saif M Mohammad, and Keith J Holyoak. 2012. Semeval-2012 task 2: Measuring degrees of relational similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation. Association for Computational Linguistics, pages 356–364.
  15. 15.Omer Levy and Yoav Goldberg. 2014a. Linguistic regularities in sparse and explicit word representations. In Proceedings of the eighteenth conference on computational natural language learning. pages 171–180.
  16. 16.Omer Levy and Yoav Goldberg. 2014b. Neural word embedding as implicit matrix factorization. In Advances in Neural Information Processing Systems. pages 2177–2185.
  17. 17.Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Linguistic knowledge and transferability of contextual representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  18. 18.Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 .
  19. 19.Thang Luong, Richard Socher, and Christopher D Manning. 2013. Better word representations with recursive neural networks for morphology. In SIGNLL Conference on Computational Natural Language Learning (CoNLL). pages 104–113.
  20. 20.Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013a. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems. pages 3111–3119.
  21. 21.Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 746–751.
  22. 22.David Mimno and Laure Thompson. 2017. The strange geometry of skip-gram with negative sampling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. pages 2873–2878.
  23. 23.Jiaqi Mu, Suma Bhat, and Pramod Viswanath. 2018. All-but-the-top: Simple and effective postprocessing for word representations. In Proceedings of the 7th International Conference on Learning Representations (ICLR).
  24. 24.Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 1532–1543.
  25. 25.Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). pages 2227–2237.
  26. 26.Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners .
  27. 27.Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations. https://openreview.net/forum?id=SJzSgnRcKX.
  28. 28.Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237 .
  29. 29.Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems. pages 3320–3328.

Citation

MLA
Ethayarajh, K. “How Contextual Are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings”. arXiv, 2019, http://arxiv.org/abs/1909.00512v1.
APA
Ethayarajh, K. (2019). How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings. arXiv. http://arxiv.org/abs/1909.00512v1
Chicago
Ethayarajh, K. 2019. “How Contextual Are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings”. arXiv. http://arxiv.org/abs/1909.00512v1.
Harvard
Ethayarajh, K. (2019) “How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1909.00512v1.
Vancouver
1. Ethayarajh K (2019) How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings. arXiv

BibTeX

@article{ethayarajh2019how,
  title = {How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings},
  author = {Ethayarajh, Kawin},
  year = {2019},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1909.00512v1},
  eprint = {1909.00512}
}
Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF
License: https://creativecommons.org/licenses/by/4.0/