Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

representational geometry

Representational geometry refers to the spatial arrangement, relative distances, and structural relationships among data points embedded within a high-dimensional vector or activation space, such as those produced by neural networks and representation learning models. Rather than examining individual dimensions, coordinates, or units in isolation, this concept focuses on how entities relate to one another through geometric properties, including pairwise distances, angles, manifold curvature, clustering patterns, and subspace orientations. By characterizing the global and local configuration of representations, representational geometry enables the quantitative analysis of how computational systems organize semantic, perceptual, or relational information, as well as how those organizational patterns transform across different layers, contexts, and architectures.

1 item

Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words

Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words

Kaitlyn Zhou, Kawin Ethayarajh, Dallas Card, Dan Jurafsky

OrganizationsStanford University

Why you should read this

Demonstrates that cosine similarity systematically underestimates the semantic similarity of high-frequency words in contextual embedding models like BERT relative to human judgment due to frequency-induced geometric distortions in embedding space.

Cosine similarity of contextual embeddings is used in many NLP tasks (e.g., QA, IR, MT) and metrics (e.g., BERTScore). Here, we uncover systematic ways in which word similarities estimated by cosine over BERT embeddings are understated and trace this effect to training data frequency. We find that relative to human judgements, cosine similarity underestimates the similarity of frequent words with other instances of the same word or other words across contexts, even after controlling for polysemy and other factors. We conjecture that this underestimation of similarity for high frequency words is due to differences in the representational geometry of high and low frequency words and provide a formal argument for the two-dimensional case.

Added

2026-10-02