Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning

Sébastien LachapelleTristan DeleuDivyat MahajanIoannis MitliagkasYoshua BengioSimon Lacoste-JulienQuentin Bertrand

article2023ICML45 citations

Proves that pairing disentangled representations with sparse predictors improves generalization, and leverages this finding to introduce an identifiable multi-task bi-level optimization method that achieves competitive few-shot classification performance using only a fraction of learned features per task.

Listen

Modern machine learning systems frequently struggle to generalize to new, unseen tasks, especially when labeled data is scarce. Learning disentangled representations—where individual components in a model's internal representation directly isolate independent, real-world factors such as color, shape, or position—is widely hypothesized to improve generalization and robustness. However, existing theoretical understanding of why and when disentanglement helps downstream tasks has remained limited, and standard unsupervised methods often fail to guarantee that true underlying factors will be recovered.

The article establishes that combining disentangled representations with sparsity-promoting predictors—using only a small subset of relevant features per task—signiﬁcantly enhances generalization in multi-task and few-shot learning environments. It formally proves a novel mathematical identifiability theorem showing that regularizing task-specific predictors toward maximal sparsity across multiple supervised tasks provably forces the shared learned representation to become disentangled, even when the underlying factors are statistically correlated or non-invertible.

To evaluate this framework, the authors implemented two-level optimization algorithms that jointly optimize a shared feature extractor while encouraging sparse feature usage for individual tasks. The approach was tested through controlled simulations, controlled experiments on the synthetic 3D Shapes image benchmark across varying levels of noise and factor correlations, and few-shot image classification on the miniImageNet benchmark using a group-sparse support vector machine formulation.

The investigation produced four primary findings. First, when downstream tasks depend on a small fraction of latent factors (e.g., 5%), combining a disentangled representation with sparse regression achieves strong predictive performance with only 25 training samples, reaching optimal accuracy at 50 samples where entangled representations require 150 samples. Second, on the 3D Shapes benchmark, the proposed sparse multi-task approach reliably recovers true factors with near-perfect correlation (scores reaching 0.96 to 0.99), maintaining robustness even when latent factors exhibit extreme correlation (0.90) and heavy noise where baseline methods degrade significantly. Third, on few-shot classification benchmarks, introducing task-level sparsity retains baseline classification accuracy (around 78–80%) while utilizing approximately 30% fewer features. Fourth, features learned through sparse multi-task objectives are selectively and dynamically reused across distinct task contexts rather than globally activated.

These results provide both theoretical and operational validation for building modular, sample-efficient artificial intelligence architectures. In deployment settings where data collection is costly, models utilizing disentangled representations and sparse task heads can reduce data acquisition requirements by up to two-thirds without sacrificing accuracy. Furthermore, by isolating individual features and promoting sparsity, models achieve greater interpretability and reduced computational footprint during downstream inference.

Organizations developing multi-task or few-shot machine learning systems should consider incorporating group-sparse regularization into their task-specific adaptation heads and feature extraction pipelines. In practice, practitioners should implement normalization layers on representations to stabilize multi-task optimization and adopt dual-formulation solvers when feature dimensions substantially exceed sample counts. Future efforts should focus on extending these identifiability guarantees to non-linear task predictors and establishing standardized unsupervised hyperparameter selection techniques, as validation on held-out tasks remains challenging due to inherent model identifiability trade-offs.

CIFAR Mila – Québec Artificial Intelligence Institute meta-learning optimization optimization problems multi-task learning disentanglement performance sparsity regularization

arXiv: 2211.14666 tristandeleu/synergies-disentanglement-sparsity

Paper: Convex multi-task feature learning, Andreas Argyriou et al. (2008). Its convex multi-task feature-learning framework and shared-representation formulation provide the foundation for understanding how the source uses sparse task predictors to identify shared features.
Paper: Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations, Francesco Locatello et al. (2018). Its impossibility result shows why unsupervised disentanglement alone cannot guarantee recovery, clarifying the need for the source’s supervised multi-task sparsity mechanism.
Paper: A Model of Inductive Bias Learning, Jonathan Baxter (2000). Its theory of learning inductive bias across related tasks supplies the generalization framework needed to follow the source’s claims about transfer to unseen tasks.
Paper: Isolating Sources of Disentanglement in Variational Autoencoders, Ricky T. Q. Chen et al. (2018). Its decomposition of the VAE objective and account of total-correlation regularization establish the disentanglement methods and concepts that the source’s identifiability results address.
Paper: beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework, Irina Higgins et al. (2016). Its β-VAE method introduces the factorized latent-representation goal and benchmark setting that help contextualize the source’s alternative, supervised route to disentanglement.

No sufficiently relevant recommendations were found.

Cover for Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning

Abstract

Although disentangled representations are often said to be beneficial for downstream tasks, current empirical and theoretical understanding is limited. In this work, we provide evidence that disentangled representations coupled with sparse task-specific predictors improve generalization. In the context of multi-task learning, we prove a new identifiability result that provides conditions under which maximally sparse predictors yield disentangled representations. Motivated by this theoretical result, we propose a practical approach to learn disentangled representations based on a sparsity-promoting bi-level optimization problem. Finally, we explore a meta-learning version of this algorithm based on group Lasso multiclass SVM predictors, for which we derive a tractable dual formulation. It obtains competitive results on standard few-shot classification benchmarks, while each task is using only a fraction of the learned representations.

1 Introduction
1.1 Contributions
1.2 Background
2 Disentanglement and Sparse Task-Specific Predictors Improve Generalization
2.1 MLE invariance to linear feature transformations
2.2 An advantage of disentangled representations
3 Sparse Multi-Task Learning for Disentanglement
3.1 Task & data generating process
3.2 Main identifiability result
3.3 Assumptions of
3.4 Tractable bilevel optimization problems for sparse multitask learning
4 Related Work
5 Experiments
5.1 Disentanglement in 3D Shapes
5.2 Sparse task-specific predictors in few-shot learning
6 Conclusion
References
A Proofs of
B Proofs of
B.1 Technical Lemmas
B.2 Proof of
B.3 Regularization in the outer problem instead of in the inner problem
B.4 What can go wrong when is violated?
B.5 holds with high probability when the number of supports is large
B.6 A distribution without density satisfying
C Optimization details
C.1 Group Lasso SVM Dual
D Experimental details
D.1 Disentangled representation coupled with sparsity regularization improves generalization
D.2 Disentanglement in 3D Shapes
D.2.1 Dataset generation
D.2.2 Metrics
D.2.3 Architecture, inner solver & hyperparameters
D.2.4 Experiments violating assumptions
D.2.5 Experiments with regularization in the outer problem
D.2.6 Visual evaluation
D.2.7 Additional metrics for disentanglement
D.3 Meta-learning experiments

Knowls

Knowl 1 — Sparse task predictors identify a disentangled shared representation
theoretical result
Let observations $x\in\mathcal X\subseteq\mathbb R^d$ have a shared ground-truth representation $f_\theta(x)\in\mathbb R^m$ . A task has coefficient matrix $W\in\mathbb R^{k\times m}$ , and its conditional label distribution is $p(y;Wf_\theta(x))$ ; the task matrices are drawn from a distribution $P_W$ , and their supports are the nonzero columns of $W$ . Consider infinitely many tasks and a learned representation $f_{\hat\theta}$ chosen to minimize expected population negative log-likelihood, where each task predictor $\hat W(W)$ is itself a population-risk minimizer subject to $\|\hat W(W)\|_{2,0}\leq\|W\|_{2,0}$ . Here $\|A\|_{2,0}$ counts the nonzero columns of a matrix.

Every minimizer is disentangled relative to $f_\theta$ : there is an invertible diagonal matrix $D$ and a permutation matrix $P$ such that $f_{\hat\theta}(x)=DPf_\theta(x)$ for every $x\in\mathcal X$ , provided the following conditions hold: the parameterization $p(y;\eta)$ is identifiable from its distribution; $f_\theta$ takes values spanning $\mathbb R^m$ ; the task matrices contain enough linearly independent rows across tasks to identify all $m$ directions; conditional on each task support, the coefficient distribution assigns zero probability to every proper hyperplane in the active coordinates; and, for each coordinate $j$ , the observed task supports that omit $j$ collectively cover every coordinate other than $j$ . The representation functions considered in the optimization must also be continuous. The result does not require independent latent factors or an invertible mapping between observations and representations; the task labels need only follow the specified linear-predictor distribution.
Knowl 2 — Sparse predictors can improve generalization on disentangled features
theoretical result
Suppose the true label distribution is $p(y;Wf_\theta(x))$ and a learned representation is linearly equivalent to the ground truth, $f_{\hat\theta}(x)=Lf_\theta(x)$ for an invertible matrix $L$ . Assume the population maximum-likelihood predictor is unique. Its coefficient matrix in the learned representation is $\hat W_\infty=WL^{-1}$ , so its fitted parameter $\hat W_\infty f_{\hat\theta}(x)$ is unchanged by the choice of invertible feature coordinates.

If the true task coefficient matrix has $\ell$ nonzero columns, imposing a limit of $\ell$ nonzero columns on the predictor does not exclude the population optimum when the representation is disentangled, since a permutation and coordinatewise rescaling preserve column sparsity. A general entangling transformation can instead make $WL^{-1}$ denser, so the same sparsity constraint can exclude the population optimum and introduce approximation error. The constraint reduces estimator complexity, and therefore can reduce estimation error for either representation; its distinctive advantage for disentangled features is avoiding that added approximation error when the true task is sparse. Consequently, sparse regularization can improve generalization most when data are limited and the task uses only a small subset of the ground-truth features; the benefit need not hold when tasks are not sufficiently sparse.
Knowl 3 — Empirical bilevel objective for learning a shared representation
model/method
For $T$ tasks with datasets $D_t$ of $n$ labeled examples each, the paper learns a shared representation $f_{\hat\theta}$ while fitting one task-specific coefficient matrix $\hat W^{(t)}\in\mathbb R^{k\times m}$ per task. Its tractable empirical objective is $\min_{\hat\theta}\,-\frac{1}{Tn}\sum_{t=1}^T\sum_{(x,y)\in D_t}\log p(y;\hat W^{(t)}f_{\hat\theta}(x))$ , subject to each $\hat W^{(t)}$ minimizing that task's average negative log-likelihood plus $\lambda_t\|W\|_{2,1}$ . The group norm is $\|W\|_{2,1}=\sum_{j=1}^m\|W_{:j}\|_2$ , which promotes feature-level sparsity by selecting or suppressing whole columns. In practice, the authors use a common regularization weight $\lambda_t=\lambda$ across tasks.

The outer objective evaluates each task's fitted predictor, while the inner optimization determines that predictor using its task data. The paper proposes computing outer hypergradients by implicit differentiation through the inner solution and then updating the representation with first-order optimization. Because simultaneous rescaling of representations and coefficients can make the sparsity penalty ineffective, the representation norm must be controlled; the authors recommend batch or layer normalization. They also consider an alternative in which the inner task fit is unregularized and the outer objective penalizes the average $\|\hat W^{(t)}\|_{2,1}$ ; experiments report results similar to the inner-regularized version.
Knowl 4 — Perfect multi-task prediction identifies features up to an invertible transform
theoretical result
Let $f_\theta(x),f_{\hat\theta}(x)\in\mathbb R^m$ be ground-truth and learned representations, and let each task's conditional label law be $p(y;Wf_\theta(x))$ with $W\in\mathbb R^{k\times m}$ drawn from $P_W$ . If the label parameterization is identifiable, the ground-truth representation takes values spanning $\mathbb R^m$ , and the collection of task matrices contains $m$ linearly independent rows, then any task predictors $\hat W(W)$ that reproduce the ground-truth label distribution for almost every task and every $x$ must satisfy $f_\theta(x)=Lf_{\hat\theta}(x)$ for an invertible matrix $L$ , and $\hat W(W)=WL$ for almost every task. Thus perfect multi-task prediction identifies the representation and task coefficients up to a shared invertible linear transformation, before any sparsity assumptions are used to restrict that transformation to a permutation and coordinatewise scaling.
Knowl 5 — Unregularized maximum likelihood is invariant to invertible feature changes
theoretical result
Let $f_{\hat\theta}(x)=Lf_\theta(x)$ for an invertible matrix $L$ , and fit an unrestricted linear predictor by maximum likelihood on the same dataset using either representation. If both maximum-likelihood solutions are unique, their fitted parameters agree for every input: $\hat W^{(\hat\theta)}f_{\hat\theta}(x)=\hat W^{(\theta)}f_\theta(x)$ . Therefore the resulting conditional label model is the same for linearly equivalent representations, whether or not the learned features are disentangled. The generalization distinction arises when predictor regularization, such as a sparsity constraint, makes the estimator depend on the feature coordinates.
Knowl 6 — Dual formulation of the group-Lasso multiclass SVM task learner
equation
For a task with $n$ training examples, $k$ classes, and $m$ representation features, let $F\in\mathbb R^{n\times m}$ contain the feature vectors as rows, and let $Y\in\mathbb R^{n\times k}$ be the one-hot label matrix. The paper's group-Lasso soft-margin multiclass SVM uses positive regularization parameters $\lambda_1$ and $\lambda_2$ for the column group norm and squared Frobenius norm, respectively. Its dual is the constrained problem $\min_{\Lambda\in\mathbb R^{n\times k}}\frac{1}{\lambda_2}\sum_{j=1}^m\|\operatorname{BST}((Y-\Lambda)^\top F_{:j},\lambda_1)\|_2^2+\langle Y,\Lambda\rangle$ , subject to $\sum_{l=1}^k\Lambda_{il}=1$ and $\Lambda_{il}\geq0$ for every example $i$ and class $l$ . Here $\Lambda$ is the dual variable, $\langle\cdot,\cdot\rangle$ is the Frobenius inner product, and block soft-thresholding is $\operatorname{BST}(a,\tau)=(1-\tau/\|a\|_2)_+a$ , with the zero-vector case mapped to zero. The primal-dual relation is $W_{:j}=\operatorname{BST}((Y-\Lambda)^\top F_{:j},\lambda_1)/\lambda_2$ . This formulation is used to solve and differentiate through the task learner in meta-learning; the authors motivate the dual particularly when the feature count is much larger than the number of examples.
Knowl 7 — Synthetic regression shows a sample-efficiency benefit for disentangled Lasso
empirical result
The paper compares Lasso and Ridge regression using frozen disentangled features or features mixed by a randomly sampled orthogonal matrix. The synthetic problem has $m=100$ ground-truth features drawn from a Gaussian with covariance entries $\Sigma_{ij}=0.9^{|i-j|}$ , labels $y=w^\top f_\theta(x)+\epsilon$ with $\epsilon\sim\mathcal N(0,0.04)$ , and a coefficient vector masked to retain varying fractions $\ell/m\in\{5\%,20\%,80\%\}$ of its entries. Training-set sizes range from 25 to 150, evaluation uses 1,000 test examples, and regularization strengths are selected by five-fold cross-validation.

For the sparsest case, disentangled-feature Lasso achieved $R^2>0.5$ with 25 training examples, while the other methods were near zero; it reached its optimal performance with 50 examples, compared with 150 for entangled-feature Lasso. The advantage diminished as the true task became less sparse and as sample size increased. Disentangled and entangled Ridge performed alike, as expected under orthogonal feature mixing because the $L_2$ penalty is invariant to that transformation.
Knowl 8 — Sparse multi-task learning disentangles correlated factors on 3D Shapes
empirical result
The authors tested the bilevel representation-learning objective on regression tasks built from the six standardized factors of the 3D Shapes dataset. Each task used 50 examples; latent vectors were sampled from distributions with varied correlations and added latent noise, and task coefficients were made sparse by independently masking Gaussian weights with Bernoulli probability $0.5$ . Tasks were generated continuously during training. A four-layer convolutional network represented images, with batch normalization on the final representation layer to control its norm. The comparison included an inner-Lasso task learner, an inner-Ridge learner, and inner-Ridge followed by linear ICA; disentanglement was assessed by mean correlation coefficient (MCC), the maximum-permutation average of absolute Pearson correlations between true and learned factors.

Inner-Lasso achieved high MCC for suitable regularization strengths across the tested correlation and noise settings. In particular, the paper reports MCC $0.99$ for uncorrelated latents with noise scale 1 and MCC $0.96$ for correlation $0.9$ with noise scale 1. Ridge combined with ICA degraded under strong factor correlations, consistent with its independence assumption, and additional latent noise also hurt that baseline. When task supports violated the theory's richness condition, inner-Lasso performed less well overall; it nevertheless retained reasonably high disentanglement for mild violations, while the no-sparsity setting performed poorly and could be surpassed by Ridge. All methods generally retained high prediction scores, consistent with identification up to a linear transformation being weaker than full disentanglement.
Knowl 9 — Group-sparse meta-learning uses fewer miniImageNet features at similar accuracy
empirical result
The paper adapted a metric-based few-shot learner by using group-Lasso-regularized multiclass SVMs as task-specific predictors and cross-entropy on held-out task examples as the outer meta-learning loss. The SVM was optimized through its dual, and the representation used a 12-layer residual network with $m=1.6\times10^4$ features. On miniImageNet, evaluation used 5-shot, 5-way classification tasks, with at most 25 training examples per task. The reported meta-validation accuracy remained similar to the dense-predictor baseline at around 30% sparsity in task predictors, with accuracy declining as sparsity increased further. Feature-use frequencies also varied across tasks rather than concentrating on a single shared subset, which the authors interpret as evidence that learned features are reused in different task contexts.
Knowl 10 — Scope and unresolved issues
limitation
The disentanglement guarantee applies to linear task-specific predictors and relies on population-level fitting, sufficiently varied tasks and representations, and a rich collection of task supports; it does not establish identifiability for the paper's few-shot meta-learning formulation. The authors identify extending the theory to potentially nonlinear task predictors as future work. They also do not solve the practical problem of selecting the sparsity hyperparameter for disentanglement: held-out predictive fit alone is not informative for this purpose because multiple representations can be observationally equivalent.

Coverage note — Supplementary visualizations, detailed proof steps, and secondary metric and optimizer ablations were omitted because they support the stated results rather than adding separate main contributions.

References

1.Ahuja, K., Shanmugam, K., Varshney, K. R., and Dhurandhar, A. Invariant risk minimization games. In Proceedings of the 37th International Conference on Machine Learning, 2020.
2.Ahuja, K., Hartford, J., and Bengio, Y. Properties from mechanisms: an equivariance perspective on identifiable representation learning. In International Conference on Learning Representations, 2022a.
3.Ahuja, K., Hartford, J., and Bengio, Y. Weakly supervised representation learning with sparse perturbations, 2022b.
4.Ahuja, K., Mahajan, D., Syrgkanis, V., and Mitliagkas, I. Towards efficient representation identification in supervised learning. In First Conference on Causal Learning and Reasoning, 2022c.
5.Argyriou, A., Evgeniou, T., and Pontil, M. Convex multitask feature learning. Machine learning, 73(3):243–272, 2008.
6.Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D. Invariant risk minimization, 2020.
7.Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
8.Bengio, Y. Gradient-based optimization of hyperparameters. Neural computation, 12(8):1889–1900, 2000.
9.Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 2013.
10.Bertinetto, L., Henriques, J. F., Torr, P. H., and Vedaldi, A. Meta-learning with differentiable closed-form solvers. 2019.
11.Bertrand, Q., Klopfenstein, Q., Blondel, M., Vaiter, S., Gramfort, A., and Salmon, J. Implicit differentiation of lasso-type models for hyperparameter optimization. In International Conference on Machine Learning, pp. 810–821. PMLR, 2020.
12.Bertrand, Q., Klopfenstein, Q., Massias, M., Blondel, M., Vaiter, S., Gramfort, A., and Salmon, J. Implicit differentiation for fast hyperparameter selection in non-smooth convex learning. JMLR, 2022.
13.Bickel, P. J., Ritov, Y., and Tsybakov, A. B. Simultaneous analysis of lasso and Dantzig selector. The Annals of statistics, 37(4):1705–1732, 2009.
14.Blondel, M., Berthet, Q., Cuturi, M., Frostig, R., Hoyer, S., Llinares-López, F., Pedregosa, F., and Vert, J.-P. Efficient and modular implicit differentiation. NeurIPS, 2022.
15.Bolte, J., Le, T., E., Pauwels, and Silveti-Falls, T. Nonsmooth implicit differentiation for machine-learning and optimization. Advances in neural information processing systems, 34:13537–13549, 2021.
16.Bolte, J., Pauwels, E., and Vaiter, S. Automatic differentiation of nonsmooth iterative algorithms. NeurIPS, 2022.
17.Bouchacourt, D., Tomioka, R., and Nowozin, S. Multilevel variational autoencoder: Learning disentangled representations from grouped observations. Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
18.Boyd, S. P., , and Vandenberghe, L. Convex optimization. Cambridge university press, 2004.
19.Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., Vander-Plas, J., Wanderman-Milne, S., and Zhang, Q. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax.
20.Brehmer, J., De Haan, P., Lippe, P., and Cohen, T. Weakly supervised causal representation learning. In Advances in Neural Information Processing Systems, 2022.
21.Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020a.
22.Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901, 2020b.
23.Burgess, C. and Kim, H. 3d shapes dataset. https://github.com/deepmind/3dshapes-dataset/, 2018.
24.Casella, G. and Berger, R. Statistical Inference. Duxbury Resource Center, 2001.
25.Chen, R. T. Q., Li, X., G., R., and Duvenaud, D. Isolating sources of disentanglement in vaes. In Advances in Neural Information Processing Systems, 2018.
26.Chen, S. S., Donoho, D. L., and Saunders, M. A. Atomic decomposition by basis pursuit. SIAM Journal on Scientific Computing, 1998.
27.Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pp. 1597–1607. PMLR, 2020.
28.Comon, P. Independent component analysis. Higher-Order Statistics, 1992.
29.Crammer, K. and Singer, Y. On the algorithmic implementation of multiclass kernel-based vector machines. Journal of machine learning research, 2(Dec):265–292, 2001.
30.Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
31.Dittadi, A., Träuble, F., Locatello, F., Wuthrich, M., Agrawal, V., Winther, O., Bauer, S., and Schölkopf, B. On the transfer of disentangled representations in realistic settings. In International Conference on Learning Representations, 2021.
32.Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021.
33.Duan, S., Matthey, L., Saraiva, A., Watters, N., Burgess, C., Lerchner, A., and Higgins, I. Unsupervised model selection for variational disentangled representation learning. In International Conference on Learning Representations, 2020.
34.Eastwood, C. and Williams, C. K. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018.
35.Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126–1135. PMLR, 2017.
36.Goyal, A. and Bengio, Y. Inductive biases for deep learning of higher-level cognition. Proc. R. Soc. A 478: 20210068, 2022.
37.Gresele, L., Kügelgen, J. V., Stimper, V., Schölkopf, B., and Besserve, M. Independent mechanism analysis, a new concept? In Advances in Neural Information Processing Systems, 2021.
38.Gribonval, R. and Lesage, S. A survey of sparse component analysis for blind source separation: principles, perspectives, and new challenges. In ESANN’06 proceedings - 14th European Symposium on Artificial Neural Networks, 2006.
39.Higgins, I., Matthey, L., Pal, A., Burgess, C. P., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017.
40.Hoerl, A. E. and Kennard, R. W. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1):55–67, 1970.
41.Hospedales, T., Antoniou, A., Micaelli, P., and Storkey, A. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(9):5149–5169, 2021.
42.Hsieh, C.-J., Chang, K.-W., Lin, C.-J., Keerthi, S. S., and Sundararajan, S. A dual coordinate descent method for large-scale linear svm. In Proceedings of the 25th international conference on Machine learning, pp. 408–415, 2008.
43.Hyvärinen, A. and Morioka, H. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. In Advances in Neural Information Processing Systems, 2016.
44.Hyvärinen, A. and Morioka, H. Nonlinear ICA of Temporally Dependent Stationary Sources. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, 2017.
45.Hyvärinen, A. and Pajunen, P. Nonlinear independent component analysis: Existence and uniqueness results. Neural Networks, 1999.
46.Hyvärinen, A., Karhunen, J., and Oja, E. Independent Component Analysis. Wiley, 2001.
47.Hyvärinen, A., Sasaki, H., and Turner, R. E. Nonlinear ica using auxiliary variables and generalized contrastive learning. In AISTATS. PMLR, 2019.
48.Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, 2015.
49.Khemakhem, I., Kingma, D., Monti, R., and Hyvärinen, A. Variational autoencoders and nonlinear ica: A unifying framework. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, 2020a.
50.Khemakhem, I., Monti, R., Kingma, D., and Hyvärinen, A. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica. In Advances in Neural Information Processing Systems, 2020b.
51.Kim, H. and Mnih, A. Disentangling by factorising. In Proceedings of the 35th International Conference on Machine Learning, 2018.
52.Klindt, D. A., Schott, L., Sharma, Y., Ustyuzhaninov, I., Brendel, W., Bethge, M., and Paiton, D. M. Towards nonlinear disentanglement in natural data with temporal sparse coding. In 9th International Conference on Learning Representations, 2021.
53.Kreutz-Delgado, K., Murray, J. F., Rao, B. D., Engan, K., Lee, T.-W., and Sejnowski, T. J. Dictionary learning algorithms for sparse representation. Neural computation, 15(2):349–396, 2003.
54.Krueger, D., Caballero, E., Jacobsen, J.-H., Zhang, A., Binas, J., Priol, R. L., Zhang, D., and Courville, A. Out-of-distribution generalization via risk extrapolation ({re}x), 2021.
55.Kumar, A., Sattigeri, P., and Balakrishnan, A. Variational inference of disentangled latent concepts from unlabeled observations. In International Conference on Learning Representations, 2018.
56.Lachapelle, S. and Lacoste-Julien, S. Partial disentanglement via mechanism sparsity. In UAI 2022 Workshop on Causal Representation Learning, 2022.
57.Lachapelle, S., Rodriguez Lopez, P., Sharma, Y., Everett, K. E., Le Priol, R., Lacoste, A., and Lacoste-Julien, S. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA. In First Conference on Causal Learning and Reasoning, 2022.
58.Lee, K., S.Maji, Ravichandran, A., and Soatto, S. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10657–10665, 2019.
59.Lippe, P., Magliacane, S., Löwe, S., Asano, Y. M., Cohen, T., and Gavves, E. iCITRIS: Causal representation learning for instantaneous temporal effects. In UAI 2022 Workshop on Causal Representation Learning, 2022a.
60.Lippe, P., Magliacane, S., Löwe, S., Asano, Y. M., Cohen, T., and Gavves, E. CITRIS: Causal identifiability from temporal intervened sequences, 2022b.
61.Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., and Bachem, O. Challenging common assumptions in the unsupervised learning of disentangled representations. In Proceedings of the 36th International Conference on Machine Learning, 2019.
62.Locatello, F., Poole, B., Raetsch, G., Schölkopf, B., Bachem, O., and Tschannen, M. Weakly-supervised disentanglement without compromises. In Proceedings of the 37th International Conference on Machine Learning, 2020a.
63.Locatello, F., Tschannen, M., Bauer, S., Rätsch, G., Schölkopf, B., and Bachem, O. Disentangling factors of variations using few labels. In International Conference on Learning Representations, 2020b. URL https://openreview.net/forum?id=SygagpEKwB.
64.Lounici, K., Pontil, M., and Tsybakov, A. B. Oracle inequalities and optimal inference under group sparsity. The Annals of statistics, 2011a.
65.Lounici, K., Pontil, M., Van De Geer, S., and Tsybakov, A. B. Oracle inequalities and optimal inference under group sparsity. The annals of statistics, 2011b.
66.Lu, C., Wu, Y., Hernández-Lobato, J. M., and Schölkopf, B. Nonlinear invariant risk minimization: A causal approach, 2021.
67.Mairal, J., Ponce, J., Sapiro, G., Zisserman, A., and Bach, F. Supervised dictionary learning. Advances in neural information processing systems, 21, 2008.
68.Mairal, J., Bach, F., Ponce, J., and Sapiro, G. Online dictionary learning for sparse coding. In Proceedings of the 26th annual international conference on machine learning, pp. 689–696, 2009.
69.Mairal, J., Bach, F., and Ponce, J. Task-driven dictionary learning. IEEE transactions on pattern analysis and machine intelligence, 34(4):791–804, 2011.
70.Malézieux, B., Moreau, T., and Kowalski, M. Dictionary and prior learning with unrolled algorithms for unsupervised inverse problems. ICLR, 2022.
71.Marcus, G., Davis, E., and Aaronson, S. A very preliminary analysis of dall-e 2. arXiv preprint arXiv:2204.13807, 2022.
72.Maurer, A., Pontil, M., and Romera-Paredes, B. Sparse coding for multitask and transfer learning. ICML’13, 2013.
73.Maurer, A., Pontil, M., and Romera-Paredes, B. The benefit of multitask representation learning. J. Mach. Learn. Res., 2016.
74.Mikolov, T., Karafiát, M., Burget, L., Cernocký, J., and Khudanpur, S. Recurrent neural network based language model. ISCA, 2010.
75.Miladinovi, D., Gondal, M. W., Schölkopf, B., Buhmann, J. M., and Bauer, S. Disentangled state space representations. arXiv preprint arXiv:1906.03255, 2019.
76.Mohri, M., Rostamizadeh, A., and Talwalkar, A. MIT Press, 2018.
77.Montero, M. L., Ludwig, C. J., Costa, R. P., Malhotra, G., and Bowers, J. The role of disentanglement in generalisation. In International Conference on Learning Representations, 2021.
78.Moran, G. E., Sridhar, D., Wang, Y., and Blei, D. Identifiable deep generative models via sparse decoding. Transactions on Machine Learning Research, 2022.
79.Oord, A., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding. Advances in Neural Information Processing Systems, 2018.
80.Pearl, J. The seven tools of causal inference, with reflections on machine learning. Commun. ACM, 2019.
81.Pedregosa, F. Hyperparameter optimization with approximate gradient. In International conference on machine learning, pp. 737–746. PMLR, 2016.
82.Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748–8763. PMLR, 2021.
83.Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
84.Richtárik, P. and Takáč, M. Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function. Mathematical Programming, 144(1):1–38, 2014.
85.Roeder, G., Metz, L., and Kingma, D. P. On linear identifiability of learned representations. In Proceedings of the 38th International Conference on Machine Learning, 2021.
86.Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., and Bengio, Y. Toward causal representation learning. Proceedings of the IEEE - Advances in Machine Learning and Deep Neural Networks, 2021.
87.Schölkopf, B. Causality for machine learning, 2019.
88.Snell, J., Swersky, K., and Zemel, R. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems, 30, 2017.
89.Taleb, A. and Jutten, C. Source separation in post-nonlinear mixtures. IEEE Transactions on Signal Processing, 1999.
90.Tibshirani, R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267–288, 1996.
91.Tseng, P. Convergence of a block coordinate descent method for nondifferentiable minimization. Journal of optimization theory and applications, 109(3):475–494, 2001.
92.van Steenkiste, S., Locatello, F., Schmidhuber, J., and Bachem, O. Are disentangled representations helpful for abstract visual reasoning? In Advances in Neural Information Processing Systems, 2019.
93.Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
94.Von Kügelgen, J., Sharma, Y., Gresele, L., Brendel, W., Schölkopf, B., Besserve, M., and Locatello, F. Self-supervised learning with data augmentations provably isolates content from style. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
95.Wainwright, M. J. and Jordan, M. I. Graphical models, exponential families, and variational inference. Found. Trends Mach. Learn., 2008.
96.Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., and Schmidt, L. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7959–7971, June 2022.
97.Wright, S. and Nocedal, J. Numerical optimization. Springer Science, 35(67-68):7, 1999.
98.Yao, W., Sun, Y., Ho, A., Sun, C., and Zhang, K. Learning temporally causal latent processes from general temporal data. In International Conference on Learning Representations, 2022.
99.Zhang, H., Zhang, Y.-F., Liu, W., Weller, A., Schölkopf, B., and Xing, E. Towards principled disentanglement for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
100.Zheng, Y., Ng, I., and Zhang, K. On the identifiability of nonlinear ICA: Sparsity and beyond. In Advances in Neural Information Processing Systems, 2022.

Citation

MLA

Lachapelle, S., et al. “Synergies Between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning”. International Conference on Machine Learning, vol. 202, 2023, pp. 18171–206, https://proceedings.mlr.press/v202/lachapelle23a.html.

APA

Lachapelle, S., Deleu, T., Mahajan, D., Mitliagkas, I., Bengio, Y., Lacoste-Julien, S., & Bertrand, Q. (2023). Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning. International Conference on Machine Learning, 202, 18171–18206. https://proceedings.mlr.press/v202/lachapelle23a.html

Chicago

Lachapelle, S., T. Deleu, D. Mahajan, et al. 2023. “Synergies Between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning”. International Conference on Machine Learning 202: 18171–206. https://proceedings.mlr.press/v202/lachapelle23a.html.

Harvard

Lachapelle, S. et al. (2023) “Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning”, International Conference on Machine Learning. PMLR, pp. 18171–18206. Available at: https://proceedings.mlr.press/v202/lachapelle23a.html.

Vancouver

1. Lachapelle S, Deleu T, Mahajan D, Mitliagkas I, Bengio Y, Lacoste-Julien S, Bertrand Q (2023) Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning. In: International Conference on Machine Learning. PMLR, pp 18171–18206

BibTeX

@InProceedings{pmlr-v202-lachapelle23a,
  title = 	 {Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning},
  author =       {Lachapelle, Sebastien and Deleu, Tristan and Mahajan, Divyat and Mitliagkas, Ioannis and Bengio, Yoshua and Lacoste-Julien, Simon and Bertrand, Quentin},
  booktitle = 	 {Proceedings of the 40th International Conference on Machine Learning},
  pages = 	 {18171--18206},
  year = 	 {2023},
  editor = 	 {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan},
  volume = 	 {202},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {23--29 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v202/lachapelle23a/lachapelle23a.pdf},
  url = 	 {https://proceedings.mlr.press/v202/lachapelle23a.html},
  abstract = 	 {Although disentangled representations are often said to be beneficial for downstream tasks, current empirical and theoretical understanding is limited. In this work, we provide evidence that disentangled representations coupled with sparse task-specific predictors improve generalization. In the context of multi-task learning, we prove a new identifiability result that provides conditions under which maximally sparse predictors yield disentangled representations. Motivated by this theoretical result, we propose a practical approach to learn disentangled representations based on a sparsity-promoting bi-level optimization problem. Finally, we explore a meta-learning version of this algorithm based on group Lasso multiclass SVM predictors, for which we derive a tractable dual formulation. It obtains competitive results on standard few-shot classification benchmarks, while each task is using only a fraction of the learned representations.}
}

Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Sparse task predictors identify a disentangled shared representation

Knowl 2 — Sparse predictors can improve generalization on disentangled features

Knowl 3 — Empirical bilevel objective for learning a shared representation

Knowl 4 — Perfect multi-task prediction identifies features up to an invertible transform

Knowl 5 — Unregularized maximum likelihood is invariant to invertible feature changes

Knowl 6 — Dual formulation of the group-Lasso multiclass SVM task learner

Knowl 7 — Synthetic regression shows a sample-efficiency benefit for disentangled Lasso

Knowl 8 — Sparse multi-task learning disentangles correlated factors on 3D Shapes

Knowl 9 — Group-sparse meta-learning uses fewer miniImageNet features at similar accuracy

Knowl 10 — Scope and unresolved issues

References

Citation

BibTeX

Source Code

Access the Paper