keyword
data-dependent regularization
Data-dependent regularization is a machine learning approach in which the regularization mechanism or penalty function adapts directly to the structure, distribution, or geometry of the training data rather than applying a fixed, data-agnostic constraint on model parameters. Unlike standard uniform regularization techniques such as isotropic weight decay, data-dependent regularizers leverage empirical properties, including feature covariance, local input neighborhoods, or gradient-based sample perturbations, to selectively constrain model complexity along specific directions in the feature space. This form of regularization can be implemented explicitly through data-adaptive loss penalties or induced implicitly through techniques such as data augmentation and adversarial training. By aligning constraints with the underlying characteristics of the input data, data-dependent regularization enhances generalization, promotes invariance to task-irrelevant variations, and helps prevent overfitting in complex learning models.
2 items
The good, the bad and the ugly sides of data augmentation: An implicit spectral regularization perspective
Chi-Heng Lin, Chiraag Kaushik, Eva L. Dyer, Vidya Muthukumar
Why you should read this
Establishes a theoretical framework that reveals how data augmentation acts as implicit spectral regularization by reshaping the covariance spectrum and adding ridge penalty, explaining why common augmentations can either aid or harm generalization across underparameterized and overparameterized regimes.
Data augmentation (DA) is a powerful workhorse for bolstering performance in modern machine learning. Specific augmentations like translations and scaling in computer vision are traditionally believed to improve generalization by generating new (artificial) data from the same distribution. However, this traditional viewpoint does not explain the success of prevalent augmentations in modern machine learning (e.g. randomized masking, cutout, mixup), that greatly alter the training data distribution. In this work, we develop a new theoretical framework to characterize the impact of a general class of DA on underparameterized and overparameterized linear model generalization. Our framework reveals that DA induces implicit spectral regularization through a combination of two distinct effects: a) manipulating the relative proportion of eigenvalues of the data covariance matrix in a training-data-dependent manner, and b) uniformly boosting the entire spectrum of the data covariance matrix through ridge regression. These effects, when applied to popular augmentations, give rise to a wide variety of phenomena, including discrepancies in generalization between over-parameterized and under-parameterized regimes and differences between regression and classification tasks. Our framework highlights the nuanced and sometimes surprising impacts of DA on generalization, and serves as a testbed for novel augmentation design.
Added
2026-10-04
Robust Optimization as Data Augmentation for Large-scale Graphs
Kezhi Kong, Guohao Li, Mucong Ding, Zuxuan Wu, Chen Zhu, Bernard Ghanem, Gavin Taylor, Tom Goldstein
Why you should read this
Proposes FLAG, a model-free adversarial data augmentation method that perturbs node features during training to improve graph neural network generalization across node classification, link prediction, and graph property tasks with minimal computational overhead.
Data augmentation helps neural networks generalize better by enlarging the training set, but it remains an open question how to effectively augment graph data to enhance the performance of GNNs (Graph Neural Networks). While most existing graph regularizers focus on manipulating graph topological structures by adding/removing edges, we offer a method to augment node features for better performance. We propose FLAG (Free Large-scale Adversarial Augmentation on Graphs), which iteratively augments node features with gradient-based adversarial perturbations during training. By making the model invariant to small fluctuations in input data, our method helps models generalize to out-of-distribution samples and boosts model performance at test time. FLAG is a general-purpose approach for graph data, which universally works in node classification, link prediction, and graph classification tasks. FLAG is also highly flexible and scalable, and is deployable with arbitrary GNN backbones and large-scale datasets. We demonstrate the efficacy and stability of our method through extensive experiments and ablation studies. We also provide intuitive observations for a deeper understanding of our method. We open source our implementation at https://github.com/devnkong/FLAG.
Added
2026-09-26