Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

unsupervised sentence representation

Unsupervised sentence representation is a natural language processing approach that maps variable-length sentences into fixed-size numerical vectors without relying on human-annotated labels. In this framework, machine learning models learn the semantic, syntactic, and contextual properties of text directly from raw, unlabeled corpora using self-supervised objectives such as contrastive learning, masked token prediction, or autoencoding. By capturing the underlying meaning of sentences, these models ensure that semantically similar sentences are mapped close to one another in the continuous embedding space. The resulting dense vector representations can then be applied to diverse downstream tasks, including semantic textual similarity measurement, information retrieval, clustering, and classification, without requiring supervised fine-tuning.

1 item

Unsupervised Sentence Representation via Contrastive Learning with Mixing Negatives

Unsupervised Sentence Representation via Contrastive Learning with Mixing Negatives

Yanzhao Zhang, Richong Zhang, Samuel Mensah, Xudong Liu, Yongyi Mao

OrganizationsAdvanced Innovation Center for Big Data and Brain ComputingBeihang UniversitySKLSDEUniversity of OttawaUniversity of Sheffield

Why you should read this

Proposes MixCSE, a contrastive sentence representation framework that overcomes vanishing gradient signals by continually generating artificial hard negative features through mixing positive and negative samples, achieving state-of-the-art results on semantic textual similarity and transfer tasks.

Unsupervised sentence representation learning is a fundamental problem in natural language processing. Recently, contrastive learning has made great success on this task. Existing contrastive learning based models usually apply random sampling to select negative examples for training. Previous work in computer vision has shown that hard negative examples help contrastive learning to achieve faster convergence and better optimization for representation learning. However, the importance of hard negatives in contrastive learning for sentence representation is yet to be explored. In this study, we prove that hard negatives are essential for maintaining strong gradient signals in the training process while random sampling negative examples is ineffective for sentence representation. Accordingly, we present a contrastive model, MixCSE, that extends the current state-of-the-art SimCSE by continually constructing hard negatives via mixing both positive and negative features. The superior performance of the proposed approach is demonstrated via empirical studies on Semantic Textual Similarity datasets and Transfer task datasets.

Added

2026-09-26