keyword
latent semantic analysis
Latent semantic analysis is a technique in natural language processing and information retrieval that analyzes relationships between a collection of documents and the terms they contain by uncovering hidden conceptual structures across a text corpus. Based on the distributional principle that words with similar meanings tend to appear in similar contexts, the method begins by constructing a term-document matrix that records word frequencies or weighted associations across texts. It then applies a mathematical dimensionality reduction method known as singular value decomposition to project the high-dimensional matrix into a lower-dimensional latent space. By representing both words and documents as vectors within this compact semantic space, the technique captures underlying conceptual associations, helps mitigate challenges such as synonymy and polysemy, and facilitates tasks such as document classification, clustering, and semantic search based on meaning rather than literal keyword matching.
11 items
Restricted Boltzmann machines for collaborative filtering
Ruslan Salakhutdinov, Andriy Mnih, Geoffrey Hinton
Why you should read this
Demonstrates how Restricted Boltzmann Machines can be effectively adapted to massive, sparse recommendation datasets like the Netflix Prize, outperforming standard matrix factorization techniques and significantly boosting accuracy when blended in ensembles.
Most of the existing approaches to collaborative filtering cannot handle very large data sets. In this paper we show how a class of two-layer undirected graphical models, called Restricted Boltzmann Machines (RBM’s), can be used to model tabular data, such as user’s ratings of movies. We present efficient learning and inference procedures for this class of models and demonstrate that RBM’s can be successfully applied to the Netflix data set, containing over 100 million user/movie ratings. We also show that RBM’s slightly outperform carefully-tuned SVD models. When the predictions of multiple RBM models and multiple SVD models are linearly combined, we achieve an error rate that is well over 6% better than the score of Netflix’s own system.
Added
2026-09-27
License
Published with permission
Improving Distributional Similarity with Lessons Learned from Word Embeddings
Omer Levy, Yoav Goldberg, Ido Dagan
Why you should read this
Demonstrates that the superior performance of neural word embeddings over traditional count-based models stems from hyperparameter optimizations rather than algorithmic differences, proving that applying these same tuning strategies to count-based methods eliminates the performance gap across semantic benchmarks.
Recent trends suggest that neural-network-inspired word embedding models outperform traditional count-based distributional models on word similarity and analogy detection tasks. We reveal that much of the performance gains of word embeddings are due to certain system design choices and hyperparameter optimizations, rather than the embedding algorithms themselves. Furthermore, we show that these modifications can be transferred to traditional distributional models, yielding similar gains. In contrast to prior reports, we observe mostly local or insignificant performance differences between the methods, with no global advantage to any single approach over the others.
Added
2026-09-25
Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors
Marco Baroni, Georgiana Dinu, Germán Kruszewski
Why you should read this
Demonstrates through an extensive empirical evaluation across multiple lexical semantics benchmarks that context-predicting word embedding models consistently outperform traditional count-based distributional semantic vectors across various parameter configurations.
Context-predicting models (more commonly known as embeddings or neural language models) are the new kids on the distributional semantics block. Despite the buzz surrounding these models, the literature is still lacking a systematic comparison of the predictive models with classic, count-vector-based distributional semantic approaches. In this paper, we perform such an extensive evaluation, on a wide range of lexical semantics tasks and across many parameter settings. The results, to our own surprise, show that the buzz is fully justified, as the context-predicting models obtain a thorough and resounding victory against their count-based counterparts.
Added
2026-09-25
Measuring praise and criticism: Inference of semantic orientation from association
Peter D. Turney, Michael L. Littman
Why you should read this
Proposes a method for automatically determining the positive or negative semantic orientation of words across diverse parts of speech by measuring their statistical association with paradigm seed words using pointwise mutual information and latent semantic analysis.
The evaluative character of a word is called its semantic orientation. Positive semantic orientation indicates praise (e.g., "honest", "intrepid") and negative semantic orientation indicates criticism (e.g., "disturbing", "superfluous"). Semantic orientation varies in both direction (positive or negative) and degree (mild to strong). An automated system for measuring semantic orientation would have application in text classification, text filtering, tracking opinions in online discussions, analysis of survey responses, and automated chat systems (chatbots). This paper introduces a method for inferring the semantic orientation of a word from its statistical association with a set of positive and negative paradigm words. Two instances of this approach are evaluated, based on two different statistical measures of word association: pointwise mutual information (PMI) and latent semantic analysis (LSA). The method is experimentally tested with 3,596 words (including adjectives, adverbs, nouns, and verbs) that have been manually labeled positive (1,614 words) and negative (1,982 words). The method attains an accuracy of 82.8% on the full test set, but the accuracy rises above 95% when the algorithm is allowed to abstain from classifying mild words.
Added
2026-09-24
Unsupervised Learning of Human Action Categories Using Spatial-Temporal Words
Juan Carlos Niebles, Hongchen Wang, Li Fei-Fei
Why you should read this
Proposes an unsupervised framework using space-time interest points and probabilistic Latent Semantic Analysis to automatically recognize and spatio-temporally localize multiple human actions in complex video sequences without manual annotations.
We present a novel unsupervised learning method for human action categories. A video sequence is represented as a collection of spatial-temporal words by extracting space-time interest points. The algorithm automatically learns the probability distributions of the spatial-temporal words and intermediate topics corresponding to human action categories. This is achieved by using a probabilistic Latent Semantic Analysis (pLSA) model. Given a novel video sequence, the model can categorize and localize the human action(s) contained in the video. We test our algorithm on two challenging datasets: the KTH human action dataset and a recent dataset of figure skating actions. Our results are on par or slightly better than the best reported results. In addition, our algorithm can recognize and localize multiple actions in long and complex video sequences containing multiple motions.
Added
2026-09-18
Reading Tea Leaves: How Humans Interpret Topic Models
Jonathan D. Chang, Jordan L. Boyd-Graber, S. Gerrish, Chong Wang, D. Blei
Why you should read this
Introduces word and topic intrusion tasks to quantitatively evaluate topic model interpretability, revealing that statistical metrics like held-out likelihood often negatively correlate with human semantic comprehension.
Probabilistic topic models are a popular tool for the unsupervised analysis of text, providing both a predictive model of future text and a latent topic representation of the corpus. Practitioners typically assume that the latent space is semantically meaningful. It is used to check models, summarize the corpus, and guide explo- ration of its contents. However, whether the latent space is interpretable is in need of quantitative evaluation. In this paper, we present new quantitative methods for measuring semantic meaning in inferred topics. We back these measures with large-scale user studies, showing that they capture aspects of the model that are undetected by previous measures of model quality based on held-out likelihood. Surprisingly, topic models which perform better on held-out likelihood may infer less semantically meaningful topics.
Added
2026-09-14
Learning Word Vectors for Sentiment Analysis
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, Christopher Potts
Why you should read this
Proposes a semi-supervised framework for learning word vectors that capture both semantic meaning and sentiment polarity, while establishing the widely used IMDb movie review dataset for sentiment classification.
Unsupervised vector-based approaches to semantics can model rich lexical meanings, but they largely fail to capture sentiment information that is central to many word meanings and important for a wide range of NLP tasks. We present a model that uses a mix of unsupervised and supervised techniques to learn word vectors capturing semantic term–document information as well as rich sentiment content. The proposed model can leverage both continuous and multi-dimensional sentiment information as well as non-sentiment annotations. We instantiate the model to utilize the document-level sentiment polarity annotations present in many online documents (e.g. star ratings). We evaluate the model using small, widely used sentiment and subjectivity corpora and find it out-performs several previously introduced methods for sentiment classification. We also introduce a large dataset of movie reviews to serve as a more robust benchmark for work in this area.
Added
2026-09-09
Unsupervised Dense Information Retrieval with Contrastive Learning
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, Edouard Grave
Why you should read this
Establishes a contrastive learning framework for unsupervised dense information retrieval that outperforms traditional methods like BM25 in zero-shot and multilingual settings, demonstrating robust cross-lingual transfer even with limited supervision.
Recently, information retrieval has seen the emergence of dense retrievers, using neural networks, as an alternative to classical sparse methods based on term-frequency. These models have obtained state-of-the-art results on datasets and tasks where large training sets are available. However, they do not transfer well to new applications with no training data, and are outperformed by unsupervised term-frequency methods such as BM25. In this work, we explore the limits of contrastive learning as a way to train unsupervised dense retrievers and show that it leads to strong performance in various retrieval settings. On the BEIR benchmark our unsupervised model outperforms BM25 on 11 out of 15 datasets for the Recall@100. When used as pre-training before fine-tuning, either on a few thousands in-domain examples or on the large MS~MARCO dataset, our contrastive model leads to improvements on the BEIR benchmark. Finally, we evaluate our approach for multi-lingual retrieval, where training data is even scarcer than for English, and show that our approach leads to strong unsupervised performance. Our model also exhibits strong cross-lingual transfer when fine-tuned on supervised English data only and evaluated on low resources language such as Swahili. We show that our unsupervised models can perform cross-lingual retrieval between different scripts, such as retrieving English documents from Arabic queries, which would not be possible with term matching methods.
Added
2026-06-07
Dense Passage Retrieval for Open-Domain Question Answering
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih
Why you should read this
Establishes that dense passage retrieval, trained with a simple dual-encoder framework on limited data, significantly outperforms traditional sparse methods like BM25, achieving new state-of-the-art results in open-domain question answering.
Open-domain question answering relies on efficient passage retrieval to select candidate contexts, where traditional sparse vector space models, such as TF-IDF or BM25, are the de facto method. In this work, we show that retrieval can be practically implemented using dense representations alone, where embeddings are learned from a small number of questions and passages by a simple dual-encoder framework. When evaluated on a wide range of open-domain QA datasets, our dense retriever outperforms a strong Lucene-BM25 system largely by 9%-19% absolute in terms of top-20 passage retrieval accuracy, and helps our end-to-end QA system establish new state-of-the-art on multiple open-domain QA benchmarks.
Added
2026-05-04
Indexing By Latent Semantic Analysis
Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, Richard Harshman
Why you should read this
Introduces Latent Semantic Analysis as a foundational technique for information retrieval that utilizes singular value decomposition to model hidden semantic structures in text, providing a robust mathematical framework for overcoming the linguistic barriers of synonymy and polysemy in document indexing.
A new method for automatic indexing and retrieval is described. The approach is to take advantage of implicit higher-order structure in the association of terms with documents ("semantic structure") in order to improve the detection of relevant documents on the basis of terms found in queries. The particular technique used is singular-value decomposition, in which a large term by document matrix is decomposed into a set of ca. 100 orthogonal factors from which the original matrix can be approximated by linear combination. Documents are represented by ca. 100 item vectors of factor weights. Queries are represented as pseudo-document vectors formed from weighted combinations of terms, and documents with supra-threshold cosine values are returned. Initial tests find this completely automatic method for retrieval to be promising.
Added
2026-02-21
A Neural Probabilistic Language Model
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin
Why you should read this
Introduces a neural approach to capture long-range word dependencies and semantic similarities between words, overcoming the fundamental limitations of traditional n-gram models that can only consider the previous one or two words and treat each word as completely distinct from similar ones.
A goal of statistical language modeling is to learn the joint probability function of sequences of words in a language. This is intrinsically difficult because of the curse of dimensionality: a word sequence on which the model will be tested is likely to be different from all the word sequences seen during training. Traditional but very successful approaches based on n-grams obtain generalization by concatenating very short overlapping sequences seen in the training set. We propose to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences. The model learns simultaneously (1) a distributed representation for each word along with (2) the probability function for word sequences, expressed in terms of these representations. Generalization is obtained because a sequence of words that has never been seen before gets high probability if it is made of words that are similar (in the sense of having a nearby representation) to words forming an already seen sentence. Training such large models (with millions of parameters) within a reasonable time is itself a significant challenge. We report on experiments using neural networks for the probability function, showing on two text corpora that the proposed approach significantly improves on state-of-the-art n-gram models, and that the proposed approach allows to take advantage of longer contexts.
Added
2025-11-18