Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

debiased contrastive learning

Debiased contrastive learning is a self-supervised machine learning approach designed to train data representations while correcting for sampling biases that arise when unlabelled examples are selected as negative pairs. In standard contrastive learning, models are trained to pull similar positive pairs closer together in the representation space and push dissimilar negative pairs farther apart, but standard random or in-batch negative sampling often inadvertently includes semantically similar instances, known as false negatives, or distorted samples. Debiased contrastive learning resolves this problem by adjusting the loss function, reweighting instances, or refining negative sample selection to statistically approximate the true distribution of dissimilar examples without requiring explicit category labels. This correction prevents the model from mistakenly repelling related items, thereby preserving semantic consistency, improving alignment, and ensuring a uniform distribution across the learned representation space.

1 item

Debiased Contrastive Learning of Unsupervised Sentence Representations

Debiased Contrastive Learning of Unsupervised Sentence Representations

Kun Zhou, Beichen Zhang, Wayne Xin Zhao, Ji-Rong Wen

OrganizationsRenmin University of China

Why you should read this

Proposes a debiased contrastive learning framework that improves unsupervised sentence embeddings by downweighting false negatives and generating optimized noise-based negative samples to overcome representation anisotropy.

Recently, contrastive learning has been shown to be effective in improving pre-trained language models (PLM) to derive high-quality sentence representations. It aims to pull close positive examples to enhance the alignment while push apart irrelevant negatives for the uniformity of the whole representation space. However, previous works mostly adopt in-batch negatives or sample from training data at random. Such a way may cause the sampling bias that improper negatives (e.g., false negatives and anisotropy representations) are used to learn sentence representations, which will hurt the uniformity of the representation space. To address it, we present a new framework DCLR (Debiased Contrastive Learning of unsupervised sentence Representations) to alleviate the influence of these improper negatives. In DCLR, we design an instance weighting method to punish false negatives and generate noise-based negatives to guarantee the uniformity of the representation space. Experiments on seven semantic textual similarity tasks show that our approach is more effective than competitive baselines. Our code and data are publicly available at the link: https://github.com/RUCAIBox/DCLR.

Added

2026-09-26