keyword

Quick-Thought Vectors

Quick-Thought Vectors are dense numerical representations of entire sentences learned through an unsupervised, discriminative framework designed to capture broad semantic and syntactic text properties. Unlike generative sentence-encoder architectures that train by reconstructing the exact words of neighboring sentences, the Quick-Thought approach treats context prediction as a contrastive classification task. In this formulation, neural encoders map sentences into continuous vector spaces and optimize the model to identify true adjacent context sentences from a pool of candidate distractor sentences based on vector similarity. By replacing computationally expensive token-level decoding with discriminative sentence comparison, this technique substantially accelerates training speed and efficiency while producing high-quality, transferable embeddings for downstream natural language processing tasks.

1 item

Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere

Tongzhou Wang, Phillip Isola

OrganizationsMassachusetts Institute of Technology

Why you should read this

Demonstrates that contrastive representation learning fundamentally optimizes feature alignment and hyperspherical uniformity, providing quantifiable metrics that can be directly optimized to match or surpass standard contrastive methods on downstream tasks.

Contrastive representation learning has been outstandingly successful in practice. In this work, we identify two key properties related to the contrastive loss: (1) alignment (closeness) of features from positive pairs, and (2) uniformity of the induced distribution of the (normalized) features on the hypersphere. We prove that, asymptotically, the contrastive loss optimizes these properties, and analyze their positive effects on downstream tasks. Empirically, we introduce an optimizable metric to quantify each property. Extensive experiments on standard vision and language datasets confirm the strong agreement between both metrics and downstream task performance. Remarkably, directly optimizing for these two metrics leads to representations with comparable or better performance at downstream tasks than contrastive learning. Project Page: this https URL Code: this https URL , this https URL

Source

https://arxiv.org/pdf/2005.10242

Added

2026-09-14