keyword
sentence representation learning
Sentence representation learning is a subfield of natural language processing focused on training computational models to transform entire sentences into dense, fixed-dimensional numerical vectors that encapsulate their semantic meaning, syntax, and contextual relationships. By projecting textual information into a continuous vector space where semantically related statements are positioned close to each other, this process enables algorithms to mathematically compare and manipulate language at the sentence level. These representations are typically acquired using supervised, self-supervised, or contrastive learning techniques that evaluate sentence contexts, transformations, or relational dependencies within larger documents. Once derived, these embeddings serve as critical components for a wide range of downstream applications, including semantic textual similarity, document summarization, information retrieval, and text classification.
2 items
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Scott Yih, Yoon Kim, James R. Glass
Why you should read this
Proposes DiffCSE, an unsupervised sentence embedding framework that improves semantic similarity performance by pairing standard contrastive learning with an auxiliary difference prediction task to make representations sensitive to meaning-altering word edits.
We propose DiffCSE, an unsupervised contrastive learning framework for learning sentence embeddings. DiffCSE learns sentence embeddings that are sensitive to the difference between the original sentence and an edited sentence, where the edited sentence is obtained by stochastically masking out the original sentence and then sampling from a masked language model. We show that DiffCSE is an instance of equivariant contrastive learning (Dangovski et al., 2021), which generalizes contrastive learning and learns representations that are insensitive to certain types of augmentations and sensitive to other “harmful” types of augmentations. Our experiments show that DiffCSE achieves state-of-the-art results among unsupervised sentence representation learning methods, outperforming unsupervised SimCSE1 by 2.3 absolute points on semantic textual similarity tasks. 2
Added
2026-09-26
HEGEL: Hypergraph Transformer for Long Document Summarization
Haopeng Zhang, Xiao Liu, Jiawei Zhang
Why you should read this
Proposes a hypergraph transformer architecture that captures high-order cross-sentence dependencies across section structures, latent topics, and keyword coreferences to improve extractive summarization for long documents.
Extractive summarization for long documents is challenging due to the extended structured input context. The long-distance sentence dependency hinders cross-sentence relations modeling, the critical step of extractive summarization. This paper proposes HEGEL, a hypergraph neural network for long document summarization by capturing high-order cross-sentence relations. HEGEL updates and learns effective sentence representations with hypergraph transformer layers and fuses different types of sentence dependencies, including latent topics, keywords coreference, and section structure. We validate HEGEL by conducting extensive experiments on two benchmark datasets, and experimental results demonstrate the effectiveness and efficiency of HEGEL.
Added
2026-09-26