Re-basin via implicit Sinkhorn differentiation

Fidel A. Guerrero-PeñaHeitor Rapela MedeirosThomas DubailMasih AminbeidokhtiEric GrangerMarco Pedersoli

article2023CVPR67 citations

Introduces a fully differentiable neural network re-basin framework using implicit Sinkhorn differentiation, enabling gradient-based optimization of weight permutations for model merging and continual learning via linear mode connectivity.

Listen

Deep neural networks are difficult to analyze and merge because their optimization landscapes contain many symmetric solutions. While models trained from different starting points often achieve functionally equivalent performance, combining them directly causes severe performance degradation due to high loss barriers between them. Aligning internal network components via permutation symmetries—known as re-basin—can eliminate these barriers and enable direct model fusion. However, existing alignment methods rely on discrete, non-differentiable algorithms that optimize layers greedily, frequently getting stuck in suboptimal configurations and remaining difficult to adapt to custom training objectives.

The article evaluates a differentiable re-basin framework using implicit Sinkhorn differentiation to find optimal component permutations across neural network layers. The objective is to demonstrate that this differentiable alignment directly enables gradient-based optimization for arbitrary loss functions, lowers interpolation barriers between independently trained models, and mitigates catastrophic forgetting in continual learning.

To evaluate this framework, the authors conducted empirical comparisons across regression and image classification benchmarks, including polynomial approximation tasks, MNIST, CIFAR-10, and Split CIFAR-100. The approach relaxes discrete permutation matrices into continuous doubly stochastic matrices, optimized using standard gradient descent paired with efficient implicit differentiation. The authors tested this method across three settings: reconstructing exact permutations on synthetic benchmarks, minimizing loss barriers along linear paths connecting two trained models, and continually learning successive tasks by fusing past and incoming model parameters with a small replay buffer.

The experiments produced several key findings. First, the proposed differentiable formulation achieved zero alignment error across all tested network depths and initializations, whereas existing greedy weight-matching algorithms frequently settled in local minima with noticeable alignment discrepancies. Second, when connecting independently trained models, the method reduced loss barriers and path areas significantly better than prior techniques; for instance, on CIFAR-10, the data-driven random-point loss reduced the connection barrier to 0.06 compared to 0.15 for the prior Straight-Through Estimator. Third, applying this re-basin approach to continual learning on Rotated MNIST yielded an accuracy of 78.14% with a low forgetting rate of 0.12, outperforming established baselines such as Average Gradient Episodic Memory at 68.47% accuracy and 0.28 forgetting.

These results demonstrate that neural networks can be merged and adapted flexibly across diverse tasks using standard gradient-based pipelines without requiring complex, hand-crafted assignment solvers. This capability provides a practical foundation for lowering compute and communication costs in distributed systems, ensembling, and lifelong machine learning systems by allowing models to be unified in weight space while preserving prior knowledge.

Organizations aiming to implement continual model updates or model merging should adopt differentiable Sinkhorn alignment as a modular alternative to discrete assignment algorithms. Practitioners should tune the stability-plasticity hyperparameter to strike the desired balance between retaining historical performance and rapidly incorporating new data domains. Where memory resources are constrained, developers must account for the higher memory consumption of Sinkhorn iterations compared to traditional Hungarian-based solvers. Additional testing on larger production architectures and specialized real-world domains remains recommended before wide enterprise deployment.

École de technologie supérieure LIVIA, Dept. of Systems Engineering implicit Sinkhorn differentiation re-basin method linear mode connectivity

arXiv: 2212.12042

Paper: Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances, Marco Cuturi (2013). Introduces entropic regularization and the Sinkhorn-Knopp matrix scaling algorithm, which form the foundational mathematical basis for the differentiable optimal transport used in the Sinkhorn re-basin network.
Paper: Computational Optimal Transport, Gabriel Peyré et al. (2018). Provides a comprehensive theoretical and algorithmic treatment of computational optimal transport, including regularized assignment problems critical to model permutation and re-basin methods.
Paper: Averaging Weights Leads to Wider Optima and Better Generalization, Pavel Izmailov et al. (2018). Establishes foundational insights into loss landscapes, weight averaging, and mode connectivity in deep neural networks that motivate re-basin alignment across solution basins.
Paper: Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence, Arslan Chaudhry et al. (2018). Presents key concepts in incremental and continual learning benchmarks that the source paper builds on to demonstrate the utility of linear mode connectivity via re-basin.
Paper: Overcoming catastrophic forgetting in neural networks, James Kirkpatrick et al. (2017). Introduces the core challenge of catastrophic forgetting in continual learning, providing the baseline context for parameter preservation evaluated in the source paper.

Paper: Equivariant Architectures for Learning in Deep Weight Spaces, Aviv Navon et al. (2023). Extends the study of permutation symmetries and functional equivalence across neural network weight spaces by designing neural architectures equivariant to weight permutations.

Cover for Re-basin via implicit Sinkhorn differentiation

Abstract

The recent emergence of new algorithms for permuting models into functionally equivalent regions of the solution space has shed some light on the complexity of error surfaces and some promising properties like mode connectivity. However, finding the permutation that minimizes some objectives is challenging, and current optimization techniques are not differentiable, which makes it difficult to integrate into a gradient-based optimization, and often leads to sub-optimal solutions. In this paper, we propose a Sinkhorn re-basin network with the ability to obtain the transportation plan that better suits a given objective. Unlike the current state-of-art, our method is differentiable and, therefore, easy to adapt to any task within the deep learning domain. Furthermore, we show the advantage of our re-basin method by proposing a new cost function that allows performing incremental learning by exploiting the linear mode connectivity property. The benefit of our method is compared against similar approaches from the literature under several conditions for both optimal transport and linear mode connectivity. The effectiveness of our continual learning method based on re-basin is also shown for several common benchmark datasets, providing experimental results that are competitive with the state-of-art. The source code is provided at https://github.com/fagp/sinkhorn-rebasin.

Keywords

implicit Sinkhorn differentiation re-basin method linear mode connectivity optimal transport loss landscape catastrophic forgetting continual learning model fusion weights matching Straight-Through Estimator

1. Introduction
2. Related Work
3. Re-basin via the Sinkhorn Operator
4. Re-basin Incremental Learning
5. Experimental Results and Analysis
5.1. Finding the Optimal Transport
5.2. Linear Mode Connectivity
5.3. Incremental Learning Application
6. Conclusion
References

Knowls

Knowl 1 — Permutation-symmetric re-basin transformation
model/method
Let $f_\theta$ be a neural network with parameters $\theta=\{W_i,b_i\}_{i=1}^h$ , where layer $i$ maps an input $z$ to $\ell_i(z)=\sigma(W_i z+b_i)$ , $W_i$ and $b_i$ are its weights and biases, and $\sigma$ is a nonlinear activation. A re-basin transformation changes the neuron ordering at each hidden layer while preserving the represented function. For permutation matrices $P_i$ , with $P_0=P_h=I$ , the transformed layer is
$\ell_i'(z)=\sigma\!\left(P_iW_iP_{i-1}^{T}z+P_ib_i\right).$
For an exactly valid permutation, the transformed network satisfies $f_{\pi_P(\theta)}(x)=f_\theta(x)$ for every input $x$ , and therefore has the same task cost. The transformation moves a model to a different, functionally equivalent region of parameter space so that it can be linearly merged with another model.
Knowl 2 — Differentiable Sinkhorn re-basin
model/method
The Sinkhorn re-basin replaces each discrete permutation matrix with a differentiable transport plan. For a score matrix $X\in\mathbb{R}^{m\times n}$ and temperature $\tau>0$ , it computes
$S_\tau(X)=\underset{P\in\Pi}{\operatorname{arg\,max}}\;\langle P,X\rangle_F+\tau h(P),$
where $\langle\cdot,\cdot\rangle_F$ is the Frobenius inner product, $h(P)=-\sum_{a,b}P_{ab}\log P_{ab}$ is entropy, and $\Pi$ is the transportation polytope with uniform row and column marginals: every row sums to $1/m$ and every column sums to $1/n$ . The Sinkhorn iterations start from $S_\tau^{(0)}(X)=\exp(X/\tau)$ and alternately normalize rows and columns:
$S_\tau^{(t+1)}(X)=T_c\!\left(T_r\!\left(S_\tau^{(t)}(X)\right)\right),$
where $T_r(A)_{ij}=A_{ij}/\sum_{k=1}^{n}A_{ik}$ and $T_c(A)_{ij}=A_{ij}/\sum_{k=1}^{m}A_{kj}$ . As $t$ increases, the iterations approach the entropy-regularized optimum. The paper differentiates this optimum using implicit Sinkhorn differentiation rather than backpropagating through all Sinkhorn iterations, using marginals $a=\mathbf{1}_m/m$ and $b=\mathbf{1}_n/n$ . If $P_i$ is a learnable score matrix, the differentiable re-based layer is
$\ell_i'(z)=\sigma\!\left(S_\tau(P_i)W_iS_\tau(P_{i-1}^{T})z+S_\tau(P_i)b_i\right).$
This construction permits gradient-based optimization of the transport plans for any differentiable objective.
Knowl 3 — Objective functions for learning the re-basin
model/method
Given a target model $\theta_A$ , a source model $\theta_B$ , and a differentiable re-basing transformation $\pi_P(\theta_B)$ , the Sinkhorn re-basin can optimize objectives that are not restricted to a linear assignment formulation. The paper uses three objectives.

The data-free weight-matching objective is the squared parameter distance
$C_{L2}(P;\theta_A,\theta_B)=\left\|\theta_A-\pi_P(\theta_B)\right\|_2^2.$
The midpoint objective directly minimizes the task cost $C$ at the center of the interpolation path:
$C_{\mathrm{Mid}}(P;\theta_A,\theta_B)=C\!\left(\frac{\theta_A+\pi_P(\theta_B)}{2}\right).$
The random-point objective samples an interpolation coefficient $\lambda\sim U(0,1)$ independently at each optimization step and minimizes
$C_{\mathrm{Rnd}}(P;\theta_A,\theta_B)=C\!\left((1-\lambda)\theta_A+\lambda\pi_P(\theta_B)\right).$
Here, $C(\theta)$ is the supervised task loss of model parameters $\theta$ . The midpoint objective can produce a path with a low center cost but high costs away from the center; random-point sampling is introduced to reduce this multimodal-path failure mode.
Knowl 4 — Re-basin continual-learning update
model/method
For a model $\theta_i$ after learning datasets $T_0,\ldots,T_i$ , the method jointly learns a transport plan $P_i$ and a residual parameter update $\delta_i$ for the next dataset $T_{i+1}$ . Its continual-learning objective is
$C_{CL}(\delta_i,P_i;\theta_i)=C\!\left(\frac{\theta_i+\pi_{P_i}(\theta_i)}{2}+\delta_i\right)+\beta\lVert\delta_i\rVert_2^2,$
where $C$ is evaluated using the current and replayed previous datasets, $\beta\geq 0$ controls residual regularization, and $\delta_i$ is a learnable vector in the parameter space. The transport plan and residual are optimized simultaneously:
$(\delta_i^*,P_i^*)=\underset{\delta_i,P_i}{\operatorname{arg\,min}}\;C_{CL}(\delta_i,P_i;\theta_i).$
The model used for the next episode is then
$\theta_{i+1}=(1-\alpha)\theta_i+\alpha\pi_{P_i}(\theta_i)+\delta_i,$
where $\alpha\in[0,1]$ controls stability versus plasticity. Values near $0.5$ emphasize the midpoint and therefore adaptation to the new dataset, whereas values near $0$ or $1$ give greater weight to the existing model or its re-based equivalent. In the reported continual-learning experiments, $\alpha=0.8$ was used to favor retention of previous knowledge.
Knowl 5 — General experimental protocol
experimental setup
The paper evaluates Sinkhorn re-basin on permutation recovery, linear mode connectivity, and continual learning. Results are reported as means and standard deviations over independent random seeds. The permutation and connectivity experiments use polynomial regression, MNIST, and CIFAR-10 with feedforward networks ranging from two to eight layers; the Sinkhorn approximation uses $t=20$ normalization iterations and temperature $\tau=1.0$ . The permutation-recovery experiment uses Adam with initial learning rate $0.1$ , at most five optimization iterations, and early stopping upon convergence. The continual-learning experiments use 20 episodes and compare against fine-tuning, elastic weight consolidation, learning without forgetting, and A-GEM. Replay-based methods receive only five examples per class. Rotated MNIST uses a one-hidden-layer network with 256 neurons; Split CIFAR-100 uses a multi-head ResNet-18, with re-basing applied only to its linear layers.
Knowl 6 — Exact recovery of artificially applied permutations
data/table
The permutation-recovery experiment gives each method a model and a randomly permuted version of that same model, so the ground-truth transport plan is known. It contains 9 configurations formed by three initializations—random weights sampled from $\mathcal{N}(0,1)$ , a model trained on a third-degree polynomial task, and a model trained on a first-degree polynomial task—and three depths with 2, 4, or 8 hidden layers. Each configuration contains 50 model pairs. The metric is the $L_1$ distance between the estimated and ground-truth re-based weights, with all values scaled by $10^3$ ; lower is better.
Could not parse LaTeX table
Sinkhorn re-basin reaches zero error in every configuration, whereas weight matching has nonzero error particularly for random initialization and shallow networks. The result supports the paper’s claim that jointly optimizing all layer transport plans avoids the local minima encountered by the greedy layer-wise weight-matching procedure.
Knowl 7 — Linear mode connectivity criterion
definition
For two models $\theta_A$ and $\theta_B$ with task cost $C$ , the linear interpolation path is $(1-\lambda)\theta_A+\lambda\theta_B$ for $\lambda\in(0,1)$ . The paper measures its excess loss above the straight interpolation of endpoint costs using the barrier
$B(\theta_A,\theta_B)=\sup_{\lambda\in(0,1)}\left[C\!\left((1-\lambda)\theta_A+\lambda\theta_B\right)-\left((1-\lambda)C(\theta_A)+\lambda C(\theta_B)\right)\right].$
A small barrier indicates linear mode connectivity: the two endpoints can be connected without a substantial loss increase. The experiments also report the area under the sampled cost curve along the same path; both barrier and area-under-curve have a lower bound of zero. Re-basin is successful when the path from $\theta_A$ to $\pi_P(\theta_B)$ has substantially smaller values than the naive path from $\theta_A$ to $\theta_B$ .
Knowl 8 — Sinkhorn objectives improve linear mode connectivity
data/table
Two independently trained networks with two hidden layers are re-based on first- and third-degree polynomial regression, MNIST, and CIFAR-10. Each comparison is repeated 50 times. The reported area under the cost curve and barrier are lower-is-better measures of the linear interpolation between the unchanged network and the re-based network.
Could not parse LaTeX table
Every re-basing method improves substantially over the naive interpolation. The data-free Sinkhorn $C_{L2}$ objective improves on weight matching on all datasets except MNIST, where the difference is not significant. The data-driven Sinkhorn objectives improve on the straight-through estimator; random-point optimization is generally strongest on the regression tasks and is comparable to midpoint optimization on CIFAR-10.
Knowl 9 — Continual-learning benchmark results
data/table
The continual-learning method is evaluated for 20 episodes on Rotated MNIST and Split CIFAR-100. Accuracy is averaged over previously encountered tasks and higher values are better; forgetting is the average decrease in accuracy on earlier tasks and lower values are better. The re-basin method uses replay with five examples per class and sets $\alpha=0.8$ .
Could not parse LaTeX table
Re-basin with replay has the highest accuracy and lowest forgetting among the continual-learning methods on both benchmarks. It remains below joint training, which has simultaneous access to all data, but substantially improves over fine-tuning, EWC, LwF, and A-GEM. The particularly low forgetting demonstrates the stability benefit of fusing the current model with its re-based counterpart while allowing a residual update toward the new data.
Knowl 10 — Memory-efficiency limitation
limitation
The paper identifies a computational trade-off: greedy Hungarian or linear-assignment approaches generally use less memory than the Sinkhorn operator. Thus, although implicit Sinkhorn differentiation provides a more flexible, differentiable, and objective-agnostic re-basing mechanism, its memory demands can be higher than those of assignment-based weight matching, especially for larger models.

Coverage note — Supplementary-only convolutional/residual-network demonstrations and the detailed $\alpha$/$\beta$ ablation are omitted because their numerical details are not provided in the supplied main paper text.

References

1.Ryan Prescott Adams and Richard S Zemel. Ranking via sinkhorn propagation. arXiv preprint arXiv:1106.1925, 2011.
2.Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git Re-Basin: Merging Models modulo Permutation Symmetries. arXiv preprint arXiv:2209.04836, 2022.
3.Aditya Kumar Akash, Sixu Li, and Nicolas García Trillos. Wasserstein Barycenter-based Model Fusion and Linear Mode Connectivity of Neural Networks. arXiv preprint arXiv:2210.06671, 2022.
4.Peter Auer, Mark Herbster, and Manfred KK Warmuth. Exponentially many local minima for single neurons. Advances in Neural Information Processing Systems, 8, 1995.
5.Frederik Benzing, Simon Schug, Robert Meier, Johannes von Oswald, Yassir Akram, Nicolas Zucchet, Laurence Aitchison, and Angelika Steger. Random initialisations performing above chance and how to find them. arXiv preprint arXiv:2209.07509, 2022.
6.Johanni Brea, Berfin Simsek, Bernd Illing, and Wulfram Gerstner. Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape. arXiv preprint arXiv:1907.02911, 2019.
7.Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient Lifelong Learning with A-GEM. In International Conference on Learning Representations, 2019.
8.Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc’Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486, 2019.
9.Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International Conference on Machine Learning, pages 1309–1318. PMLR, 2018.
10.Marvin Eisenberger, Aysim Toker, Laura Leal-Taixe, Florian Bernard, and Daniel Cremers. A Unified Framework for Implicit Sinkhorn Differentiation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 509–518, 2022.
11.Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks. In International Conference on Learning Representations, 2022.
12.Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pages 3259–3269. PMLR, 2020.
13.Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in Neural Information Processing Systems, 31, 2018.
14.James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017.
15.Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Rong Ge, and Sanjeev Arora. Explaining landscape connectivity of low-cost solutions for multilayer nets. Advances in Neural Information Processing Systems, 32, 2019.
16.Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems, 2018.
17.Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935–2947, 2017.
18.Vincenzo Lomonaco, Lorenzo Pellegrini, Andrea Cossu, Antonio Carta, Gabriele Graffieti, Tyler L. Hayes, Matthias De Lange, Marc Masana, Jary Pomponi, Gido van de Ven, Martin Mundt, Qi She, Keiland Cooper, Jeremy Forest, Eden Belouadah, Simone Calderara, German I. Parisi, Fabio Cuzzolin, Andreas Tolias, Simone Scardapane, Luca Antiga, Subutai Amhad, Adrian Popescu, Christopher Kanan, Joost van de Weijer, Tinne Tuytelaars, Davide Bacciu, and Davide Maltoni. Avalanche: an End-to-End Library for Continual Learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2nd Continual Learning in Computer Vision Workshop, 2021.
19.Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989.
20.Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investigation of the role of pre-training in lifelong learning. arXiv preprint arXiv:2112.09153, 2021.
21.Gonzalo Mena, David Belanger, Scott Linderman, and Jasper Snoek. Learning Latent Permutations with Gumbel-Sinkhorn Networks. In International Conference on Learning Representations, 2018.
22.Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear Mode Connectivity in Multitask and Continual Learning. In International Conference on Learning Representations, 2021.
23.Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Understanding the role of training regimes in continual learning. Advances in Neural Information Processing Systems, 33:7308–7320, 2020.
24.Vaishnavh Nagarajan and J Zico Kolter. Uniform convergence may be unable to explain generalization in deep learning. Advances in Neural Information Processing Systems, 32, 2019.
25.Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285, 1990.
26.Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33:22045–22055, 2020.
27.Johannes von Oswald, Christian Henning, Benjamin F. Grewe, and Joao Sacramento. Continual learning with hypernetworks. In International Conference on Learning Representations, 2020.
28.Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. arXiv preprint arXiv:2002.06440, 2020.

Citation

MLA

Peña, F. A. G., et al. “Re-basin via Implicit Sinkhorn Differentiation”. arXiv, 2022, http://arxiv.org/abs/2212.12042v1.

APA

Peña, F. A. G., Medeiros, H. R., Dubail, T., Aminbeidokhti, M., Granger, E., & Pedersoli, M. (2022). Re-basin via implicit Sinkhorn differentiation. arXiv. http://arxiv.org/abs/2212.12042v1

Chicago

Peña, F. A. G., H. R. Medeiros, T. Dubail, M. Aminbeidokhti, E. Granger, and M. Pedersoli. 2022. “Re-basin via Implicit Sinkhorn Differentiation”. arXiv. http://arxiv.org/abs/2212.12042v1.

Harvard

Peña, F.A.G. et al. (2022) “Re-basin via implicit Sinkhorn differentiation”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2212.12042v1.

Vancouver

1. Peña FAG, Medeiros HR, Dubail T, Aminbeidokhti M, Granger E, Pedersoli M (2022) Re-basin via implicit Sinkhorn differentiation. arXiv

BibTeX

@article{pena2022basin,
  title = {Re-basin via implicit Sinkhorn differentiation},
  author = {Peña, Fidel A. Guerrero and Medeiros, Heitor Rapela and Dubail, Thomas and Aminbeidokhti, Masih and Granger, Eric and Pedersoli, Marco},
  year = {2022},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2212.12042v1},
  eprint = {2212.12042}
}

Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: IEEE

Re-basin via implicit Sinkhorn differentiation

Abstract

Keywords

Table of Contents

Knowls

Knowl 1 — Permutation-symmetric re-basin transformation

Knowl 2 — Differentiable Sinkhorn re-basin

Knowl 3 — Objective functions for learning the re-basin

Knowl 4 — Re-basin continual-learning update

Knowl 5 — General experimental protocol

Knowl 6 — Exact recovery of artificially applied permutations

Knowl 7 — Linear mode connectivity criterion

Knowl 8 — Sinkhorn objectives improve linear mode connectivity

Knowl 9 — Continual-learning benchmark results

Knowl 10 — Memory-efficiency limitation

References

Citation

BibTeX

Access the Paper