The Numerical Stability of Hyperbolic Representation Learning

Gal MishneZhengchao WanYusu WangSheng Yang

article2023ICML67 citations

Analyzes the representation and optimization trade-offs between Poincaré and Lorentz models under standard floating-point precision, presenting a Euclidean parametrization that prevents numerical instabilities in hyperbolic machine learning.

Listen

Hyperbolic machine learning has become an essential technique for analyzing hierarchical and tree-like data across domains such as natural language processing, recommendation systems, and genomics. Hyperbolic space naturally accommodates branching structures with low distortion because its volume expands exponentially. However, this same geometric property creates severe numerical instability during computational modeling. In standard 64-bit floating-point arithmetic, existing implementations routinely encounter vanishing gradients, rounding limits, and invalid calculations (NaN errors), forcing systems to constrain learning parameters or halt entirely.

The article systematically evaluates the numerical representation limits and optimization dynamics of the two most common hyperbolic frameworks—the Poincaré ball and the Lorentz model. It demonstrates mathematically and empirically why standard models fail, introduces a robust Euclidean parametrization to resolve these trade-offs, and extends this framework to hyperbolic support vector machines for classification tasks.

To conduct this evaluation, the authors performed theoretical mathematical analyses of coordinate capacities and gradient behaviors in 64-bit precision. They supplemented this theory with empirical validation on eight simulated tree-embedding datasets and six real-world classification benchmarks, including image datasets (CIFAR-10, Fashion-MNIST) and multiple single-cell RNA biological datasets. Performance was evaluated across metric distortion, gradient norms, embedding spread, classification accuracy, and macro F1 scores.

The findings establish that under 64-bit arithmetic, the Poincaré model has a wider representation capacity radius of approximately 38 from the origin before points collapse to the boundary, compared to a radius of approximately 19 for the Lorentz model. Conversely, the Lorentz model is substantially superior in optimization; the Poincaré model suffers from severe gradient vanishing near the boundary because its gradient update terms scale at second-order decay (10^-2k) versus first-order decay (10^-k) in Lorentz. The proposed Euclidean feature parametrization successfully combines the benefits of both by eliminating numerical capacity boundaries entirely while maintaining first-order optimization dynamics comparable to Lorentz. In empirical tree embeddings, the Euclidean approach achieved lower average distortion (reaching 1.0019–1.0292) and substantially larger embedding diameters (around 8.8–19.8) compared to the restricted Poincaré embeddings (around 4.2–6.4). Finally, applying this parametrization to hyperbolic hyperplanes removed non-convex constraints, creating a reformulated classifier (LSVMPP) that outperformed standard Euclidean, Poincaré, and Lorentz baselines across almost all benchmarks (e.g., reaching 89.49% on Fashion-MNIST and 62.64% on the Paul dataset).

These results demonstrate that practitioner difficulties with hyperbolic learning stem from structural flaws in coordinate optimization rather than data quality. By utilizing Euclidean parametrizations, engineering teams can eliminate catastrophic training collapses, avoid artificial thresholding hacks, and improve model accuracy on complex hierarchical data without incurring custom ultra-high-precision hardware overhead.

Engineering and research teams should adopt Euclidean parametrizations as the default training proxy for hyperbolic representation learning and hyperbolic support vector machines. Additionally, practitioners employing multi-class scaling should use hyperbolic signed distance transformations (arcsinh adjustments) to calibrate prediction confidence effectively. Before broad deployment, development teams should conduct pilots on larger-scale networks, as accelerated momentum methods in hyperbolic spaces still require further investigation.

The article's conclusions are supported with high confidence by both formal proofs and consistent multi-domain experiments. However, practitioners should note that computing certain derived functions, such as pairwise hyperbolic distances between distant points, may still encounter standard precision limits, warranting careful boundary checks in production environments.

Harvard University University of California, San Diego floating-point arithmetic formal proofs optimization hyperbolic representation learning numerical stability hyperbolic neural networks

arXiv: 2211.00181 yangshengaa/stable-hyperbolic

Paper: Poincaré Embeddings for Learning Hierarchical Representations, Maximilian Nickel et al. (2017). Read this foundational treatment of Poincaré-ball embeddings first to understand the model whose numerical capacity and optimization failures the source analyzes.

No sufficiently relevant recommendations were found.

Cover for The Numerical Stability of Hyperbolic Representation Learning

Abstract

The hyperbolic space is widely used for representing hierarchical datasets due to its ability to embed trees with small distortion. However, this property comes at a price of numerical instability such that training hyperbolic learning models will sometimes lead to catastrophic NaN problems, encountering unrepresentable values in floating point arithmetic. In this work, we analyze the limitations of two popular models for the hyperbolic space, namely, the Poincaré ball and the Lorentz model. We find that, under the 64 bit arithmetic system, the Poincaré ball has a relatively larger capacity than the Lorentz model for correctly representing points. However, the Lorentz model is superior to the Poincaré ball from the perspective of optimization, which we theoretically validate. To address these limitations, we identify one Euclidean parametrization of the hyperbolic space which can alleviate these issues. We further extend this Euclidean parametrization to hyperbolic hyperplanes and demonstrate its effectiveness in improving the performance of hyperbolic SVM.

Topics

floating-point arithmetic formal proofs optimization parameterization recommendation recommendation systems representation learning standard model support vector trade

Keywords

hyperbolic representation learning numerical stability hyperbolic neural networks Lorentz SVM hyperbolic embeddings Riemannian gradient descent hyperbolic geometry Riemannian optimization hierarchical representations

1. Introduction
2. Preliminary
3. Comparing Lorentz and Poincaré Models
3.1. Optimization
4. Euclidean Parametrization of Hyperbolic Space
4.1. Feature Parametrization
4.2. Hyperplane Parametrization
4.2.1. A New Formulation of Hyperbolic SVM
5. Experiments
5.1. Hyperbolic Embeddings
5.2. Hyperbolic SVM Models
6. Discussions
Acknowledgements
References
A. Proofs
A.1. Proof of Proposition 3.1
A.2. Proof of Proposition 3.2
A.3. Proof of Proposition 3.3
A.4. Proof of Lemma 3.4
A.5. Proof of Theorem 3.5
A.6. Proof of Theorem 4.2
A.7. Derivation of Equation (12)
B. Tree Optimization
B.1. Synthetic Data Generating Processing
B.2. Full Results
C. More on SVM
C.1. Overview of SVM in Binary Classification
C.2. Multiclass Classification Using SVM: Platt Scaling
C.2.1. Adaptation of Platt Scaling on Hyperbolic Spaces
C.3. Hyperbolic SVM Implementation and Hyperparameters
C.4. Details of Synthetic Data Generation
C.5. Full Results

Knowls

Knowl 1 — Float64 representation capacity differs sharply between the Poincaré and Lorentz models
theoretical result
Under 64-bit floating-point arithmetic, the largest reliably representable radius from the origin is approximately $38$ in the Poincaré ball and $19$ in the Lorentz model. For a Poincaré point $x$ with Euclidean norm $\|x\|=1-10^{-k}$ , its hyperbolic distance from the origin is $d(0,x)=\ln(10)k+\ln(2)+O(10^{-k})$ . In Float64, $1-10^{-k}$ rounds to $1$ when $10^{-k}\lesssim 2^{-53}\approx 10^{-16}$ , giving a maximum useful $k$ of about $16$ and radius about $38$ . For a Lorentz point $x=(x_0,x_1,\ldots,x_n)$ with $x_0=10^k$ , the distance from the Lorentz origin is $d(\bar 0,x)=\ln(10)k+\ln(2)+O(10^{-2k})$ . Around $x_0=10^8$ , floating-point arithmetic cannot distinguish $x_0^2-1$ from $x_0^2$ , compromising the defining Minkowski norm constraint and yielding the approximate radius limit $19$ .

The two limitations have different operational character. In the Poincaré ball, points beyond the threshold round to the boundary, so subsequent manifold operations are unavailable; this is a hard limit. In the Lorentz model, the time coordinate can become numerically unverifiable while the spatial coordinates remain available for further computations, making the limit comparatively soft. Equivalently, measuring capacity in the tangent space at the origin, the exponential maps can represent radial tangent vectors only up to approximately $38$ for the Poincaré model and $19$ for the Lorentz model under this arithmetic.
Knowl 2 — Poincaré-coordinate optimization suffers stronger gradient vanishing than Lorentz-coordinate optimization
theoretical result
Let $f$ be a differentiable objective on hyperbolic space, represented in the Poincaré ball by $f$ and in the Lorentz model by its isometric counterpart. Exact Riemannian gradient descent in the two models gives corresponding updates under the isometry, but the coordinate magnitudes used in floating-point computation differ. If a Poincaré point $x$ satisfies $\|x\|=1-\delta$ for small $\delta>0$ , then the Euclidean coordinate magnitude of its Poincaré Riemannian gradient is $\Omega(\delta^2\|\nabla f(x)\|)$ , whereas the Lorentz-coordinate gradient magnitude is $O(\|\nabla f(x)\|)$ .

For a radial example, take $x=(1-10^{-k},0,\ldots,0)$ , assume $\nabla f(x)=(\partial_1 f(x),0,\ldots,0)$ with $\partial_1 f(x)<0$ , let $E=\|\nabla f(x)\|$ , and use learning rate $\eta>0$ . The respective one-step updates have expansions
$\exp_x(-\eta\nabla^{D}f(x))=(1-10^{-k}+O(\eta E10^{-2k}),0,\ldots,0),$ $\exp_y(-\eta\nabla^{L}g(y))=10^k\big(1-\tfrac12 10^{-k}+\eta E10^{-k}+O(\eta E10^{-2k}),\;1-\tfrac12 10^{-k}+\eta E10^{-k}+O(\eta E10^{-2k}),\;0,\ldots,0\big),$
where $y$ is the Lorentz point corresponding to $x$ , and $D$ and $L$ denote the Poincaré and Lorentz models. For bounded $E$ and $\eta=1$ , the Poincaré update adds a term of order $10^{-2k}$ to a coordinate near $1$ , while the Lorentz update changes coordinates at relative order $10^{-k}$ . At $k=8$ , the Poincaré update is about Float64 rounding precision and can be lost; the Lorentz update is less affected by gradient vanishing. Thus, although the Poincaré model has greater point-representation capacity, much of its outer region cannot be effectively used by gradient optimization.
Knowl 3 — A Euclidean tangent-space parametrization represents hyperbolic features by unconstrained vectors
model/method
For a hyperbolic point represented by $z\in\mathbb{R}^n$ , the paper parametrizes corresponding Poincaré-ball and Lorentz-model features by the exponential maps at their origins:
$F_D(z)=\tanh\!\left(\frac{\|z\|}{2}\right)\frac{z}{\|z\|}, \qquad F_L(z)=\left(\cosh(\|z\|),\;\sinh(\|z\|)\frac{z}{\|z\|}\right).$
At $z=0$ , both expressions are defined by continuity. The Poincaré origin is $0$ and the Lorentz origin is $\bar 0=(1,0,\ldots,0)$ . These maps identify the two model representations of the same hyperbolic point and preserve radial distance: for $x=F_D(z)$ and $y=F_L(z)$ , $d_D(0,x)=d_L(\bar 0,y)=\|z\|$ . The paper uses the unconstrained Euclidean vector $z$ as the trainable feature representation, avoiding the finite-radius constraint on the stored feature coordinates in the two manifold models. This parametrization is a diffeomorphic coordinate system, not a representation preserving the Riemannian metric in all directions; numerical evaluation of hyperbolic quantities derived from these coordinates can still be problematic.
Knowl 4 — Euclidean parametrization mitigates Poincaré gradient vanishing
theoretical result
Let $h(z)=f(F_D(z))$ be a hyperbolic objective expressed using the Euclidean feature parametrization $F_D$ , and consider the radial point $x=(1-\delta,0,\ldots,0)$ with $\delta=10^{-k}$ and $z=F_D^{-1}(x)=(2\operatorname{arctanh}(1-\delta),0,\ldots,0)$ . Under the radial-gradient assumptions $\nabla f(x)=(\partial_1f(x),0,\ldots,0)$ and $\partial_1f(x)<0$ , set $E=\|\nabla f(x)\|$ . The Euclidean gradient satisfies $\|\nabla h(z)\|=\Omega(\delta E)$ , rather than the $\Omega(\delta^2 E)$ scaling of the Poincaré Riemannian gradient. A gradient step with learning rate $\eta$ has first coordinate
$z_1-\eta(\nabla h(z))_1=\ln(10)k+\ln(2)+(\eta E-\tfrac12)10^{-k}+O(10^{-2k}).$
The Jacobian of the exponential-map parametrization cancels one power of the small boundary factor responsible for Poincaré gradient vanishing. For points within the Lorentz model's practical Float64 range, the Euclidean-coordinate update has behavior similar to Lorentz optimization and is less prone to vanishing than direct Poincaré optimization.
Knowl 5 — Hyperbolic hyperplanes admit an unconstrained Euclidean parametrization
model/method
In the Lorentz model, write a point as $x=(x_0,x_r)\in\mathbb{R}^{n+1}$ , where $x_r\in\mathbb{R}^n$ , and use the Minkowski product $[u,v]=-u_0v_0+\langle u_r,v_r\rangle$ . For parameters $a\in\mathbb{R}$ and $z\in\mathbb{R}^n$ , the paper parametrizes a hyperplane by transporting a tangent vector from the Lorentz origin to a point $p$ :
$p=\left(\cosh(a),\sinh(a)\frac{z}{\|z\|}\right), \qquad w=\left(\sinh(a)\|z\|,\cosh(a)z\right).$
The resulting hyperplane is
$\widetilde H_{z,a}=\left\{x\in L^n:\cosh(a)\langle z,x_r\rangle=\sinh(a)\|z\|x_0\right\}.$
Here $p$ is at distance $|a|$ from the origin in the direction of $z$ , and $w$ has Lorentz tangent norm $\|z\|$ . The hyperplane equation remains well-defined for $z=0$ , where it degenerates to the whole space. Unlike optimizing directly over a Lorentz tangent vector, this parametrization places no constraint such as $[w,w]>0$ on the optimization variables.
Knowl 6 — LSVMPP reformulates Lorentz SVM using the Euclidean hyperplane coordinates
model/method
For Lorentz-model training examples $x_i=(x_{i0},x_{ir})\in L^n$ with binary labels $y_i\in\{-1,1\}$ , the paper substitutes the Euclidean hyperplane parameters $z\in\mathbb{R}^n$ and $a\in\mathbb{R}$ into the soft-margin Lorentz SVM objective. The resulting LSVMPP optimization problem is
$\min_{z,a}\;\frac12\|z\|^2+C\sum_{i=1}^N l_L\!\left(y_i\left[\sinh(a)\|z\|x_{i0}-\cosh(a)\langle z,x_{ir}\rangle\right]\right),$
where $N$ is the number of examples, $C>0$ controls the misclassification penalty, and $l_L(u)=\max(0,\operatorname{arcsinh}(1)-\operatorname{arcsinh}(u))$ . This removes the nonconvex feasibility constraint on the Lorentz hyperplane vector, although the resulting objective remains nonconvex. For classification probability calibration, the paper also uses the normed signed hyperplane-distance score $\operatorname{arcsinh}(-[w,x]/\|w\|_L)$ as input to Platt scaling, rather than using the raw Minkowski inner product alone.

Knowl 7 — Tree embeddings show that Euclidean-parametrized and Lorentz training outperform direct Poincaré training

data/table

Eight simulated trees were embedded using direct Poincaré optimization ( $D^2$ ), direct Lorentz optimization ( $L^2$ ), or Euclidean optimization of tangent-space coordinates with hyperbolic distances ( $E^2$ ). The raw baseline measures Euclidean distances between the original tree coordinates. Each tree was initialized, centered, and normalized to a unit square. Training minimized the squared discrepancy between normalized tree and embedding pairwise distances, using learning rate $1$ for $30{,}000$ epochs; the manifold models used Riemannian SGD and the Euclidean model used SGD. Average distortion $\delta$ is the product of mean contraction and expansion ratios, max distortion $\delta^{\max}$ uses the corresponding supremum, and $d$ is embedding diameter. Lower distortion is better; larger diameter indicates a more spread-out embedding.

The Euclidean and Lorentz models have comparable results, with Euclidean optimization usually slightly better; both generally improve over the Poincaré model. In particular, Poincaré embeddings have much smaller diameters, consistent with their optimization getting stuck before using the available hyperbolic range.

Tree	Manifold	$\delta$	$\delta^{\max}$	$d$
1	raw	1.1954	10.6062	2.8284
1	$D^2$	1.0462	4.0546	4.8740
1	$L^2$	1.0176	2.4511	10.1827
1	$E^2$	1.0157	2.3158	10.9019
2	raw	1.1186	14.1421	2.2361
2	$D^2$	1.0589	7.1435	4.9757
2	$L^2$	1.0180	3.2803	10.6719
2	$E^2$	1.0134	2.7408	11.4504
3	raw	1.0511	3.0000	2.0396
3	$D^2$	1.0321	3.3830	6.3564
3	$L^2$	1.0190	2.5506	9.6830
3	$E^2$	1.0174	2.4350	10.2328
4	raw	1.1539	19.0000	2.8284
4	$D^2$	1.0754	14.1074	5.0453
4	$L^2$	1.0421	8.2933	10.4028
4	$E^2$	1.0292	6.8504	12.7596
5	raw	1.3349	12.7279	1.4142
5	$D^2$	1.0176	2.0321	4.2523
5	$L^2$	1.0029	1.2829	7.3850
5	$E^2$	1.0019	1.1997	8.8504
6	raw	1.5080	88.8552	1.4142
6	$D^2$	1.1243	50.4524	4.3168
6	$L^2$	1.0392	21.9948	8.9204
6	$E^2$	1.0269	22.3743	9.8190
7	raw	1.2108	17.7132	0.8321
7	$D^2$	1.2145	8.6454	5.0657
7	$L^2$	1.0166	3.7655	19.1703
7	$E^2$	1.0100	1.7825	19.8481
8	raw	1.1421	10.7807	0.9220
8	$D^2$	1.0324	9.7552	4.5670
8	$L^2$	1.0157	9.2790	8.8324
8	$E^2$	1.0148	8.4500	9.1712

Knowl 8 — Hyperbolic SVM evaluation compares four classifiers across synthetic and real data
experimental setup
The classification experiments compare Euclidean SVM (ESVM), Lorentz SVM (LSVM), Poincaré SVM with a precomputed tangent reference point (PSVM), and the proposed parametrized Lorentz SVM (LSVMPP). The six synthetic datasets comprise three Gaussian mixtures and three explicit trees. Each Gaussian-mixture dataset contains 1,200 points in the Poincaré disk, formed by mapping two-dimensional Euclidean samples through the hyperbolic feature parametrization; the number of equally sized labeled clusters is $k\in\{3,5,10\}$ , with centroids sampled from $\mathcal N(0,1.5I_2)$ and noise from $\mathcal N(0,I_2)$ . For each explicit-tree dataset, a node is selected at random, its descendants form the positive class, and all other nodes form the negative class.

The six real datasets are CIFAR-10, Fashion-MNIST, Paul myeloid progenitors (19 classes), Olsson single-cell RNA sequencing (8 classes), Krumsiek simulated myeloid progenitors (11 classes), and Moignard blood-cell development (7 classes). Their points were embedded in the curvature-1 Poincaré disk. The evaluation used one-vs-all classifiers with Platt scaling; each fixed train-test split was run five times, with accuracy and macro F1 reported. The CIFAR-10, Fashion-MNIST, and Olsson splits followed the established split used by the PSVM evaluation; other real datasets used stratified 75%/25% splits. ESVM and PSVM performed best with $C=5$ and learning rate $0.001$ ; LSVM and LSVMPP generally used $C=0.5$ , learning rate around $10^{-10}$ , and 500 epochs.

Knowl 9 — LSVMPP improves results on most, but not all, real and synthetic SVM datasets

empirical result

On the six real datasets, LSVMPP with the arcsinh distance-based Platt score has the highest accuracy and macro F1 on CIFAR-10, Fashion-MNIST, Paul, and Krumsiek. PSVM leads on Olsson, while ESVM leads on Moignard. The following values are mean accuracy (percent) and macro F1, reported as mean $\pm$ standard deviation:

Dataset	Method	Accuracy (%)	Macro F1
CIFAR-10	ESVM	91.88 $\pm$ 0.00	0.9191 $\pm$ 0.00
	LSVM	91.88 $\pm$ 0.00	0.9189 $\pm$ 0.00
	PSVM	91.81 $\pm$ 0.00	0.9182 $\pm$ 0.00
	LSVMPP (raw)	91.94 $\pm$ 0.00	0.9195 $\pm$ 0.00
	LSVMPP (arcsinh)	91.96 $\pm$ 0.00	0.9197 $\pm$ 0.00
Fashion-MNIST	ESVM	86.37 $\pm$ 0.00	0.8665 $\pm$ 0.00
	LSVM	71.59 $\pm$ 0.07	0.6588 $\pm$ 0.08
	PSVM	86.57 $\pm$ 0.00	0.8665 $\pm$ 0.00
	LSVMPP (raw)	89.35 $\pm$ 0.00	0.8939 $\pm$ 0.00
	LSVMPP (arcsinh)	89.49 $\pm$ 0.00	0.8955 $\pm$ 0.00
Paul	ESVM	55.05 $\pm$ 0.00	0.4073 $\pm$ 0.00
	LSVM	58.36 $\pm$ 0.07	0.4517 $\pm$ 0.00
	PSVM	55.25 $\pm$ 0.00	0.3802 $\pm$ 0.00
	LSVMPP (raw)	62.55 $\pm$ 0.14	0.4579 $\pm$ 0.01
	LSVMPP (arcsinh)	62.64 $\pm$ 0.05	0.5024 $\pm$ 0.00
Olsson	ESVM	72.72 $\pm$ 0.00	0.4922 $\pm$ 0.00
	LSVM	81.82 $\pm$ 0.00	0.7542 $\pm$ 0.00
	PSVM	88.63 $\pm$ 0.00	0.8793 $\pm$ 0.00
	LSVMPP (raw)	80.91 $\pm$ 0.45	0.7142 $\pm$ 0.03
	LSVMPP (arcsinh)	84.09 $\pm$ 0.00	0.8429 $\pm$ 0.00
Krumsiek	ESVM	82.19 $\pm$ 0.00	0.6770 $\pm$ 0.00
	LSVM	85.62 $\pm$ 0.39	0.6933 $\pm$ 0.92
	PSVM	84.06 $\pm$ 0.00	0.6908 $\pm$ 0.00
	LSVMPP (raw)	83.75 $\pm$ 0.28	0.6403 $\pm$ 0.01
	LSVMPP (arcsinh)	86.25 $\pm$ 0.00	0.7079 $\pm$ 0.00
Moignard	ESVM	67.78 $\pm$ 0.00	0.5934 $\pm$ 0.00
	LSVM	64.88 $\pm$ 0.38	0.5502 $\pm$ 0.22
	PSVM	60.77 $\pm$ 0.00	0.5167 $\pm$ 0.00
	LSVMPP (raw)	65.77 $\pm$ 0.13	0.5671 $\pm$ 0.00
	LSVMPP (arcsinh)	65.22 $\pm$ 0.63	0.5719 $\pm$ 0.02

The pattern is not uniform on synthetic data either: LSVMPP reaches 100.00% accuracy and 1.0000 macro F1 on one explicit-tree dataset, and performs strongly on several others, but other methods tie or lead on some synthetic cases. The authors therefore report an overall advantage on most tested datasets, not universal dominance.

Knowl 10 — The Euclidean parametrization does not remove numerical instability from all hyperbolic computations
limitation
The paper's Euclidean feature coordinates avoid a finite-radius limit on the stored parameter vector, but this does not guarantee numerical stability for every computation induced by hyperbolic geometry. In particular, evaluating quantities such as hyperbolic distances between parametrized points may still produce numerical problems. The parametrization also does not preserve the full Riemannian structure, so Euclidean optimization is a proxy for optimizing hyperbolic objectives rather than Riemannian gradient descent itself.

Coverage note — Proof derivations, background geometry, visualization-only material, and secondary implementation details were omitted; the main capacity, optimization, parametrization, SVM, and experimental contributions are represented.

References

1.Becigneul, G. and Ganea, O.-E. Riemannian adaptive optimization methods. In International Conference on Learning Representations, 2018.
2.Bonnabel, S. Stochastic gradient descent on Riemannian manifolds. IEEE Transactions on Automatic Control, 58 (9):2217–2229, 2013.
3.Chamberlain, B. P., Hardwick, S. R., Wardrope, D. R., Dzogang, F., Daolio, F., and Vargas, S. Scalable hyperbolic recommender systems. arXiv preprint arXiv:1902.08648, 2019.
4.Chen, W., Han, X., Lin, Y., Zhao, H., Liu, Z., Li, P., Sun, M., and Zhou, J. Fully hyperbolic neural networks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5672–5686, 2022.
5.Chien, E., Pan, C., Tabaghi, P., and Milenkovic, O. Highly scalable and provably accurate classification in Poincaré balls. In 2021 IEEE International Conference on Data Mining (ICDM), pp. 61–70. IEEE, 2021.
6.Cho, H., DeMeo, B., Peng, J., and Berger, B. Large-margin classification in hyperbolic space. In The 22nd international conference on artificial intelligence and statistics, pp. 1832–1840. PMLR, 2019.
7.Ganea, O., Becigneul, G., and Hofmann, T. Hyperbolic neural networks. Advances in neural information processing systems, 31, 2018.
8.Gao, S., Mishne, G., and Scheinost, D. Poincaré embedding reveals edge-based functional networks of the brain. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2020, pp. 448–457. Springer, 2020.
9.Graham, R. L. An efficient algorithm for determining the convex hull of a finite planar set. Info. Pro. Lett., 1:132–133, 1972.
10.Guo, Y., Wang, X., Chen, Y., and Yu, S. X. Clipped hyperbolic classifiers are super-hyperbolic classifiers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11–20, 2022.
11.Hamilton, L. and Moitra, A. A no-go theorem for robust acceleration in the hyperbolic plane. Advances in Neural Information Processing Systems, 34:3914–3924, 2021.
12.Khrulkov, V., Mirvakhabova, L., Ustinova, E., Oseledets, I., and Lempitsky, V. Hyperbolic image embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6418–6428, 2020.
13.Kochurov, M., Karimov, R., and Kozlukov, S. Geoopt: Riemannian optimization in pytorch. arXiv preprint arXiv:2005.02819, 2020.
14.Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
15.Krumsiek, J., Marr, C., Schroeder, T., and Theis, F. J. Hierarchical differentiation of myeloid progenitors is encoded in the transcription factor network. PloS one, 6(8):e22649, 2011.
16.Law, M., Liao, R., Snell, J., and Zemel, R. Lorentzian distance learning for hyperbolic representations. In International Conference on Machine Learning, pp. 3672–3681. PMLR, 2019.
17.Lopez, F. and Strube, M. A fully hyperbolic neural model for hierarchical multi-class classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 460–475, 2020.
18.Lopez, F., Heinzerling, B., and Strube, M. Fine-grained entity typing in hyperbolic space. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pp. 169–180, 2019.
19.Mathieu, E., Le Lan, C., Maddison, C. J., Tomioka, R., and Teh, Y. W. Continuous hierarchical representations with Poincaré variational auto-encoders. Advances in neural information processing systems, 32, 2019.
20.Moignard, V., Woodhouse, S., Haghverdi, L., Lilly, A. J., Tanaka, Y., Wilkinson, A. C., Buettner, F., Macaulay, I. C., Jawaid, W., Diamanti, E., et al. Decoding the regulatory network of early blood development from single-cell gene expression measurements. Nature biotechnology, 33(3):269–276, 2015.
21.Nickel, M. and Kiela, D. Poincaré embeddings for learning hierarchical representations. Advances in neural information processing systems, 30, 2017.
22.Nickel, M. and Kiela, D. Learning continuous hierarchies in the Lorentz model of hyperbolic geometry. In International Conference on Machine Learning, pp. 3779–3788. PMLR, 2018.
23.Olsson, A., Venkatasubramanian, M., Chaudhri, V. K., Aronow, B. J., Salomonis, N., Singh, H., and Grimes, H. L. Single-cell analysis of mixed-lineage states leading to a binary cell fate choice. Nature, 537(7622):698–702, 2016.
24.Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in pytorch. 2017.
25.Paul, F., Arkin, Y., Giladi, A., Jaitin, D. A., Kenigsberg, E., Keren-Shaul, H., Winter, D., Lara-Astiaso, D., Gury, M., Weiner, A., et al. Transcriptional heterogeneity and lineage commitment in myeloid progenitors. Cell, 163 (7):1663–1677, 2015.
26.Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011.
27.Peng, W., Varanka, T., Mostafa, A., Shi, H., and Zhao, G. Hyperbolic deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
28.Platt, J. et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61–74, 1999.
29.Sala, F., De Sa, C., Gu, A., and Re, C. Representation tradeoffs for hyperbolic embeddings. In International conference on machine learning, pp. 4460–4469. PMLR, 2018.
30.Sarkar, R. Low distortion Delaunay embedding of trees in hyperbolic plane. In International Symposium on Graph Drawing, pp. 355–366. Springer, 2011.
31.Shimizu, R., Mukuta, Y., and Harada, T. Hyperbolic neural networks++. In International Conference on Learning Representations, 2020.
32.Skopek, O., Ganea, O.-E., and Becigneul, G. Mixed-curvature variational autoencoders. In International Conference on Learning Representations, 2019.
33.Ungar, A. A. Hyperbolic trigonometry and its application in the Poincaré ball model of hyperbolic geometry. Computers & Mathematics with Applications, 41(1-2):135–147, 2001.
34.Wilson, B. and Leimeister, M. Gradient descent in hyperbolic space. arXiv preprint arXiv:1805.08207, 2018.
35.Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
36.Yu, T. and De Sa, C. M. Numerically accurate hyperbolic embeddings using tiling-based models. Advances in Neural Information Processing Systems, 32, 2019.
37.Zhu, Y., Zhou, D., Xiao, J., Jiang, X., Chen, X., and Liu, Q. Hypertext: Endowing fasttext with hyperbolic geometry. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1166–1171, 2020.

Citation

MLA

Mishne, G., et al. “The Numerical Stability of Hyperbolic Representation Learning”. International Conference on Machine Learning, vol. 202, 2023, pp. 24925–49, https://proceedings.mlr.press/v202/mishne23a.html.

APA

Mishne, G., Wan, Z., Wang, Y., & Yang, S. (2023). The Numerical Stability of Hyperbolic Representation Learning. International Conference on Machine Learning, 202, 24925–24949. https://proceedings.mlr.press/v202/mishne23a.html

Chicago

Mishne, G., Z. Wan, Y. Wang, and S. Yang. 2023. “The Numerical Stability of Hyperbolic Representation Learning”. International Conference on Machine Learning 202: 24925–49. https://proceedings.mlr.press/v202/mishne23a.html.

Harvard

Mishne, G. et al. (2023) “The Numerical Stability of Hyperbolic Representation Learning”, International Conference on Machine Learning. PMLR, pp. 24925–24949. Available at: https://proceedings.mlr.press/v202/mishne23a.html.

Vancouver

1. Mishne G, Wan Z, Wang Y, Yang S (2023) The Numerical Stability of Hyperbolic Representation Learning. In: International Conference on Machine Learning. PMLR, pp 24925–24949

BibTeX

@InProceedings{pmlr-v202-mishne23a,
  title = 	 {The Numerical Stability of Hyperbolic Representation Learning},
  author =       {Mishne, Gal and Wan, Zhengchao and Wang, Yusu and Yang, Sheng},
  booktitle = 	 {Proceedings of the 40th International Conference on Machine Learning},
  pages = 	 {24925--24949},
  year = 	 {2023},
  editor = 	 {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan},
  volume = 	 {202},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {23--29 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v202/mishne23a/mishne23a.pdf},
  url = 	 {https://proceedings.mlr.press/v202/mishne23a.html},
  abstract = 	 {The hyperbolic space is widely used for representing hierarchical datasets due to its ability to embed trees with small distortion. However, this property comes at a price of numerical instability such that training hyperbolic learning models will sometimes lead to catastrophic NaN problems, encountering unrepresentable values in floating point arithmetic. In this work, we analyze the limitations of two popular models for the hyperbolic space, namely, the Poincaré ball and the Lorentz model. We find that, under the 64-bit arithmetic system, the Poincaré ball has a relatively larger capacity than the Lorentz model for correctly representing points. However, the Lorentz model is superior to the Poincaré ball from the perspective of optimization, which we theoretically validate. To address these limitations, we identify one Euclidean parametrization of the hyperbolic space which can alleviate these issues. We further extend this Euclidean parametrization to hyperbolic hyperplanes and demonstrate its effectiveness in improving the performance of hyperbolic SVM.}
}

Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

The Numerical Stability of Hyperbolic Representation Learning

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Float64 representation capacity differs sharply between the Poincaré and Lorentz models

Knowl 2 — Poincaré-coordinate optimization suffers stronger gradient vanishing than Lorentz-coordinate optimization

Knowl 3 — A Euclidean tangent-space parametrization represents hyperbolic features by unconstrained vectors

Knowl 4 — Euclidean parametrization mitigates Poincaré gradient vanishing

Knowl 5 — Hyperbolic hyperplanes admit an unconstrained Euclidean parametrization

Knowl 6 — LSVMPP reformulates Lorentz SVM using the Euclidean hyperplane coordinates