keyword
nonnegative matrix factorization
Nonnegative matrix factorization is an unsupervised dimensionality reduction and data analysis technique in linear algebra and machine learning that decomposes a matrix containing only non-negative numbers into a product of lower-rank factor matrices that are also strictly non-negative. Because the non-negativity constraint forbids negative values and cancellations, the decomposition yields an additive, parts-based representation where individual components combine without subtraction to reconstruct the original data. This property makes the resulting latent features highly interpretable, leading to wide application in fields such as document clustering, topic modeling, computer vision, gene expression analysis, and audio signal processing.
2 items
Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
Noam Shazeer, Mitchell Stern
Why you should read this
Introduces Adafactor, a memory-efficient adaptive optimizer that tracks factored row and column statistics instead of full second-moment matrices, matching Adam's training performance on large Transformers while drastically cutting optimizer memory overhead.
In several recently proposed stochastic optimization methods (e.g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients. Maintaining these per-parameter second-moment estimators requires memory equal to the number of parameters. For the case of neural network weight matrices, we propose maintaining only the per-row and per-column sums of these moving averages, and estimating the per-parameter second moments based on these sums. We demonstrate empirically that this method produces similar results to the baseline. Secondly, we show that adaptive methods can produce larger-than-desired updates when the decay rate of the second moment accumulator is too slow. We propose update clipping and a gradually increasing decay rate scheme as remedies. Combining these methods and dropping momentum, we achieve comparable results to the published Adam regime in training the Transformer model on the WMT 2014 English-German machine translation task, while using very little auxiliary storage in the optimizer. Finally, we propose scaling the parameter updates based on the scale of the parameters themselves.
Added
2026-09-25
Orthogonal nonnegative matrix t-factorizations for clustering
C. Ding, Tao Li, Wei Peng, Haesun Park
Why you should read this
Establishes a rigorous mathematical foundation and convergent update algorithms for orthogonal three-factor nonnegative matrix factorization, enabling simultaneous, interpretable co-clustering of rows and columns in complex data matrices.
Currently, most research on nonnegative matrix factorization (NMF) focus on 2-factor X = FG^T factorization. We provide a systematic analysis of 3-factor X = FSG^T NMF. While unconstrained 3-factor NMF is equivalent to unconstrained 2-factor NMF, constrained 3-factor NMF brings new features to constrained 2-factor NMF. We study the orthogonality constraint because it leads to rigorous clustering interpretation. We provide new rules for updating F,S,G and prove the convergence of these algorithms. Experiments on 5 datasets and a real world case study are performed to show the capability of bi-orthogonal 3-factor NMF on simultaneously clustering rows and columns of the input data matrix. We provide a new approach of evaluating the quality of clustering on words using class aggregate distribution and multi-peak distribution. We also provide an overview of various NMF extensions and examine their relationships.
Added
2026-09-25