keyword

uniformity loss

Uniformity loss is an objective function in representation learning designed to encourage normalized feature vectors to be distributed evenly across the surface of a unit hypersphere. By penalizing the clustering of representations and maximizing pairwise distances between data points, this metric helps prevent representation collapse and ensures the latent space preserves maximal information about the input data. Typically formulated as the logarithm of the average pairwise Gaussian potential between feature embeddings, uniformity loss acts as a repulsive force across distinct instances and is commonly paired with alignment objectives in contrastive learning to yield well-separated, generalizable representations for downstream tasks.

1 item

Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere

Tongzhou Wang, Phillip Isola

OrganizationsMassachusetts Institute of Technology

Why you should read this

Demonstrates that contrastive representation learning fundamentally optimizes feature alignment and hyperspherical uniformity, providing quantifiable metrics that can be directly optimized to match or surpass standard contrastive methods on downstream tasks.

Contrastive representation learning has been outstandingly successful in practice. In this work, we identify two key properties related to the contrastive loss: (1) alignment (closeness) of features from positive pairs, and (2) uniformity of the induced distribution of the (normalized) features on the hypersphere. We prove that, asymptotically, the contrastive loss optimizes these properties, and analyze their positive effects on downstream tasks. Empirically, we introduce an optimizable metric to quantify each property. Extensive experiments on standard vision and language datasets confirm the strong agreement between both metrics and downstream task performance. Remarkably, directly optimizing for these two metrics leads to representations with comparable or better performance at downstream tasks than contrastive learning. Project Page: this https URL Code: this https URL , this https URL

Source

https://arxiv.org/pdf/2005.10242

Added

2026-09-14