Back

Why do tree-based models still outperform deep learning on typical tabular data?

Léo GrinsztajnEdouard OyallonGaël Varoquaux

article2022NeurIPS2,014 citations

Establishes a comprehensive 45-dataset benchmark showing that tree-based models consistently outperform deep learning on tabular data, while identifying key inductive bias differences—such as handling irregular functions and uninformative features—to guide future neural architecture design.

Listen

The article investigates why tree-based ensemble models continue to outperform modern deep learning architectures on typical tabular datasets, despite the major breakthroughs deep learning has achieved in image and language processing. Tabular data remains the primary format across business, finance, and industrial applications, yet practitioners face conflicting claims regarding whether novel neural network designs can surpass established methods like gradient boosted trees and random forests. The article addresses this uncertainty by establishing a standardized benchmark to rigorously evaluate competing models and by uncovering the specific data properties and architectural biases that explain why performance gaps persist.

To establish a fair comparison, the authors curated a benchmark of 45 diverse, real-world tabular datasets and implemented a thorough evaluation procedure that accounts for the computational budget and variability of hyperparameter tuning across 20,000 compute hours. They evaluated standard and specialized deep learning models, including Multi-Layer Perceptrons, Residual Networks, and Transformer-based models, against standard tree-based models such as XGBoost and Random Forests across both classification and regression tasks on medium-sized datasets of approximately 10,000 samples.

The benchmark demonstrates that tree-based models consistently maintain state-of-the-art predictive performance, outperforming deep learning methods across all hyperparameter tuning budgets while requiring substantially less computation time. The performance gap does not vanish with extensive hyperparameter tuning, nor is it primarily caused by the presence of categorical variables, as tree models retain a clear lead on datasets containing strictly numerical features. Through controlled empirical transformations, the authors identified three primary causes for this gap: deep neural networks struggle to fit the irregular, non-smooth target functions typical of tabular data; standard neural networks are significantly hindered by uninformative features that commonly occur in tabular settings; and rotational invariance in neural network architectures impairs their ability to leverage the meaningful, individual orientations of tabular features.

These findings provide immediate practical clarity for operational decision-making, indicating that organizations should prioritize tree-based models as the primary default for tabular machine learning tasks. Choosing tree-based models yields higher predictive accuracy, lower training costs, faster deployment timelines, and reduced hardware demands compared to deep learning pipelines. For research teams developing tabular neural networks, the results demonstrate that architectures must explicitly incorporate tabular-specific inductive biases, such as breaking rotational invariance through feature-level embeddings and improving robustness to noisy, irrelevant features.

Future research should expand this benchmarking approach to very large and small sample regimes, multi-class classification, and broader scenarios involving missing values and high-cardinality categorical data. While confidence in these conclusions is high for medium-sized tabular datasets, practitioners should note that the analysis focused on curated, independent, and identically distributed data regimes, and the performance trade-offs may vary on massive datasets or when end-to-end integration with unstructured data is required.

INRIASorbonne Universitédeep learningrandom forestsregularization parameterstabular deep learningtree-based modelstabular data benchmark
arXiv: 2207.08815LeoGrin/tabular-benchmark
  • Paper: Tabular Data: Deep Learning is Not All You Need, Ravid Shwartz-Ziv et al. (2021). This benchmark directly spurred the debate on deep learning versus tree ensembles on tabular datasets by showing that gradient-boosted trees consistently outperform recent tabular neural networks.
  • Paper: TabNet: Attentive Interpretable Tabular Learning, Sercan Ö. Arik et al. (2019). It introduces TabNet, one of the primary deep learning architectures evaluated and benchmarked as a baseline against tree ensembles in the source paper.
  • Paper: XGBoost: A Scalable Tree Boosting System, Tianqi Chen et al. (2016). It introduces XGBoost, the foundational scalable gradient tree boosting framework that serves as the dominant baseline in the source study's tabular evaluations.
  • Paper: CatBoost: unbiased boosting with categorical features, Liudmila Prokhorenkova et al. (2018). It details CatBoost's categorical handling and ordered boosting mechanisms, which constitute key state-of-the-art tree ensemble baselines compared in the source benchmarks.
  • Paper: Random Forests, Leo Breiman (2001). It provides the seminal theory and algorithmic formulation of Random Forests, establishing the fundamental tree-based ensemble architecture benchmarked throughout the paper.
  • Paper: A comparative analysis of gradient boosting algorithms, Candice Bentéjac et al. (2019). It provides the empirical comparison and hyperparameter tuning methodology across XGBoost, LightGBM, and CatBoost that underpins tabular benchmarking standards.
  • Paper: Greedy function approximation: A gradient boosting machine, Jerome H. Friedman (2001). It establishes the foundational mathematical formulation of gradient boosting with decision trees upon which all modern tabular boosting systems rely.
  • Paper: Entity Embeddings of Categorical Variables, Cheng Guo et al. (2016). It establishes the standard methodology for mapping categorical features to continuous embeddings in neural networks for tabular data.
  • Paper: Do we need hundreds of classifiers to solve real world classification problems?, Manuel Fernández Delgado et al. (2014). It provides the large-scale cross-dataset evaluation framework demonstrating the historical dominance of tree-based ensembles across real-world tabular classification problems.
  • Paper: Accurate predictions on small data with a tabular foundation model, Noah Hollmann et al. (2025). It addresses the challenges and inductive bias gaps highlighted by the source by proposing TabPFN, a pre-trained tabular foundation model that outperforms gradient-boosted trees on small-to-medium datasets without per-dataset training.
Cover for Why do tree-based models still outperform deep learning on typical tabular data?

Abstract

While deep learning has enabled tremendous progress on text and image datasets, its superiority on tabular data is not clear. We contribute extensive benchmarks of standard and novel deep learning methods as well as tree-based models such as XGBoost and Random Forests, across a large number of datasets and hyperparameter combinations. We define a standard set of 45 datasets from varied domains with clear characteristics of tabular data and a benchmarking methodology accounting for both fitting models and finding good hyperparameters. Results show that tree-based models remain state-of-the-art on medium-sized data (~10K samples) even without accounting for their superior speed. To understand this gap, we conduct an empirical investigation into the differing inductive biases of tree-based models and neural networks. This leads to a series of challenges which should guide researchers aiming to build tabular-specific neural network: 1. be robust to uninformative features, 2. preserve the orientation of the data, and 3. be able to easily learn irregular functions. To stimulate research on tabular architectures, we contribute a standard benchmark and raw data for baselines: every point of a 20 000 compute hours hyperparameter search for each learner.

Topics

deep learningrandom forestsregularization parameterskernel parameterconvolutional neural networksnetwork architecture

Keywords

tabular deep learningtree-based modelstabular data benchmarktabular datasetsinductive biasesuninformative featurescategorical featureshyperparameter tuning

Table of Contents

  • 1 Introduction
  • 2 Related work
  • 3 A benchmark for tabular learning
  • 3.1 45 reference tabular datasets
  • 3.2 Removing side issues
  • 3.3 A procedure to benchmark models with hyperparameter selection
  • 3.4 Aggregating results across datasets
  • 3.5 Data preparation
  • 4 Tree-based models still outperform deep learning on tabular data.
  • 4.1 Models benchmarked
  • 4.2 Results
  • 5 Empirical investigation: why do tree-based models still outperform deep learning on tabular data?
  • 5.1 Methodology: uncovering inductive biases
  • 5.2 Finding 1: Neural networks are biased to overly smooth solutions
  • 5.3 Finding 2: Uninformative features affect more MLP-like neural networks
  • 5.4 Finding 3: Data are non invariant by rotation, so should be learning procedures
  • 6 Discussion and conclusion
  • Acknowledgments and Disclosure of Funding
  • References
  • Checklist

Knowls

  1. Knowl 1 — Empirical Superiority of Tree-Based Models Over Tabular Deep Learning

    empirical result

    On medium-sized tabular datasets (standardized to 10,00010,000 training samples), tree-based ensemble models consistently outperform both classical and tabular-specific deep learning architectures across binary classification and regression tasks. In extensive benchmarks across 45 datasets, tree-based models (XGBoost, RandomForest, and GradientBoostingTrees) achieve higher normalized test accuracy and test R2R^2 scores than deep learning models (MLP, ResNet, SAINT, and FT_Transformer).

    This superiority holds across all hyperparameter search budgets: the default configurations of tree-based models outperform default deep neural networks, and the performance gap persists even after extensive hyperparameter tuning (over 100 random search iterations). Furthermore, tree-based methods achieve these results with substantially lower computational cost per training iteration on CPUs compared to neural networks on GPUs.

  2. Knowl 2 — Tabular Benchmark Dataset Selection and Curation Protocol

    experimental setup

    To evaluate tabular algorithms without confounding factors, a benchmark suite of 45 real-world and realistic simulated datasets is curated using the following inclusion and preprocessing criteria:

    1. Heterogeneity: Features must represent distinct quantities rather than homogeneous multi-channel sensor or image signals.
    2. Dimensionality: The ratio of features dd to samples nn must satisfy d/n<1/10d/n < 1/10, with total feature count d<500d < 500, sample count n≥3 000n \ge 3\,000, and feature count d≥4d \ge 4.
    3. Non-Triviality: Datasets where a baseline model (maximum score of a single decision tree and a linear/logistic regression) comes within 5% relative score difference of default ResNet and default HistGradientBoosting are excluded as overly easy.
    4. Non-Determinism: Datasets with deterministic rule-based targets (e.g., chess or poker) and non-i.i.d. stream/time-series data are excluded.
    5. Standardization of Confounders: Training sets are truncated to 10,00010,000 samples for medium-scale benchmarks. Missing values are eliminated by removing heavily missing columns and subsequent incomplete rows. Multiclass targets are converted to balanced binary classification using the two most frequent classes. Categorical features with cardinality >20> 20 are removed, and numerical features with <10< 10 unique values are dropped (or converted to categorical if cardinality equals 2).

  3. Knowl 3 — Tabular Hyperparameter Tuning and Metric Aggregation Protocol

    algorithm

    To measure model generalization as a function of hyperparameter exploration budget while controlling for search variance, the benchmarking evaluation operates as follows:

    Input: Model family MM, Dataset D=(Xtrain,Xval,Xtest)D = (X_{\text{train}}, X_{\text{val}}, X_{\text{test}}), Hyperparameter search space H\mathcal{H}, Total budget N≈400N \approx 400, Repetitions K=15K = 15
    Output: Expected performance trajectory S(n)S(n) for n∈{1,…,N}n \in \{1, \dots, N\}
    Sample NN hyperparameter configurations h1,…,hNh_1, \dots, h_N from H\mathcal{H}, setting h1h_1 to default hyperparameters
    for i=1i = 1 to NN do
        Train M(hi)M(h_i) on XtrainX_{\text{train}}
        Evaluate score viv_i on XvalX_{\text{val}} and score tit_i on XtestX_{\text{test}}
    end for
    for k=1k = 1 to KK do
        Generate a random permutation πk\pi_k of indices {1,…,N}\{1, \dots, N\} such that πk(1)=1\pi_k(1) = 1
        for n=1n = 1 to NN do
            Identify best validation index i∗=arg⁡max⁡j∈{1,…,n}vπk(j)i^* = \arg\max_{j \in \{1, \dots, n\}} v_{\pi_k(j)}
            Record test performance Pk(n)=tπk(i∗)P_{k}(n) = t_{\pi_k(i^*)}
        end for
    end for
    Compute S(n)=1K∑k=1KPk(n)S(n) = \frac{1}{K} \sum_{k=1}^K P_k(n)
    return S(n)S(n)

    To aggregate test performance across diverse datasets, metrics (accuracy for classification, R2R^2 for regression) are affine-normalized between 00 and 11 across models on each dataset. The lower anchor (00) is set to the 10th percentile test error for classification and the 50th percentile test error for regression (with negative regression scores clipped to 00), and the upper anchor (11) is set to the top-performing model's score.

  4. Knowl 4 — Target Function Smoothness Bias in Tabular Neural Networks

    empirical result

    Neural networks struggle to fit the irregular, high-frequency target functions typical of tabular data due to their implicit spectral bias toward low-frequency (smooth) solutions, whereas tree-based models naturally learn piece-wise constant functions.

    To demonstrate this difference, training targets yy are smoothed using a Gaussian kernel smoother:

    yi′=∑jyjexp⁡(−12σ2(xi−xj)TΣ−1(xi−xj))∑jexp⁡(−12σ2(xi−xj)TΣ−1(xi−xj))y'_i = \frac{\sum_{j} y_j \exp\left(-\frac{1}{2\sigma^2} (x_i - x_j)^T \Sigma^{-1} (x_i - x_j)\right)}{\sum_{j} \exp\left(-\frac{1}{2\sigma^2} (x_i - x_j)^T \Sigma^{-1} (x_i - x_j)\right)}

    where Σ\Sigma is the empirical feature covariance matrix, xix_i are Gaussianized features, and σ≥0\sigma \ge 0 is the smoothing lengthscale (with σ=0\sigma = 0 denoting unperturbed training data).

    As the smoothing lengthscale σ\sigma increases on the training set, the test accuracy of tree-based ensembles (GradientBoostingTrees, RandomForest) drops sharply, whereas the test accuracy of neural networks (MLP, ResNet, FT_Transformer) remains virtually unchanged. This confirms that the generalization advantage of tree-based models stems from capturing non-smooth, irregular patterns in the target function that neural networks fail to learn.

  5. Knowl 5 — MLP Sensitivity to Uninformative Tabular Features

    empirical result

    Tabular datasets contain a substantial proportion of uninformative features. Decision tree ensembles naturally isolate informative variables, whereas Multi-Layer Perceptrons (MLPs) and ResNets exhibit high sensitivity to uninformative dimensions.

    Empirical observations on numerical classification datasets show:

    1. Removing up to 50% of features sorted in ascending order of Random Forest feature importance causes negligible degradation in Gradient Boosting Tree (GBT) accuracy. GBTs trained solely on the bottom 20% least important features perform close to random chance, confirming that these features are uninformative rather than merely redundant.
    2. Removing uninformative features progressively narrows the performance gap between MLPs/ResNets and tree-based models.
    3. Adding synthetic uninformative features (xnoise∼N(0,I)x_{\text{noise}} \sim \mathcal{N}(0, I) uncorrelated with both the target and other features) causes a steep decline in MLP and ResNet test accuracy, while tree-based models and tokenized architectures (FT_Transformer) remain largely unaffected.

  6. Knowl 6 — Tabular Coordinate Basis Alignment and Detriment of Rotational Invariance

    empirical result

    A machine learning procedure is rotationally invariant if applying an orthogonal matrix U∈O(d)U \in \mathcal{O}(d) to input features X↦XUX \mapsto X U on both training and test sets leaves the resulting predictions unchanged. MLPs and ResNets are rotationally invariant, whereas decision trees and feature-tokenized transformers (e.g., FT_Transformer) operate coordinate-wise and are non-rotationally invariant.

    Applying random orthogonal rotations to tabular datasets reverses model rankings: the test performance of tree-based ensembles and FT_Transformers drops drastically, while ResNet performance remains invariant, making ResNets superior to tree models on rotated data.

    Because tabular features carry individual semantic meaning (e.g., age, income) aligned with coordinate axes, rotational invariance is detrimental. In the presence of irrelevant features, rotationally invariant algorithms have sample complexity scaling at least linearly with the number of irrelevant features because the rotation mixes informative and uninformative features, whereas coordinate-aligned models can discard uninformative axes independently.

  7. Knowl 7 — Residual Tabular Performance Gap on Purely Numerical Data

    empirical result

    Although handling categorical variables is often hypothesized as the primary obstacle for neural networks on tabular datasets, tree-based models maintain their performance advantage when evaluated strictly on numerical features.

    Comparing benchmarks containing only numerical features against benchmarks containing both numerical and categorical features shows that the performance gap between tree-based ensembles (XGBoost, GBT, RandomForest) and neural networks (MLP, ResNet, FT_Transformer, SAINT) is slightly wider when categorical features are present. However, most of the performance deficit remains on purely numerical datasets, demonstrating that the gap is fundamentally driven by inductive biases regarding target smoothness, feature selection, and coordinate orientation rather than categorical encodings.

  8. Knowl 8 — Limitations of the Medium-Sized Tabular Benchmark

    limitation

    The benchmark and inductive bias analysis exhibit several deliberate boundary conditions:

    1. Sample Regime: Focuses predominantly on medium-sized training sets (truncated at 10,00010,000 samples); very small datasets (<3,000<3,000 samples) and large datasets (>50,000>50,000 samples) are not comprehensively covered.
    2. Task Scope: Classification analysis is limited to binarized tasks; multi-class classification is not evaluated.
    3. Data Preprocessing Constraints: Missing values and high-cardinality categorical features (>20>20 unique values) were removed to avoid confounding model architecture evaluation with imputation or high-cardinality encoding techniques.
    4. Metric Scope: Evaluations rely exclusively on deterministic performance metrics (accuracy for classification, R2R^2 for regression) and do not assess probabilistic prediction quality or uncertainty calibration.

Coverage note — No substantial contributed material from the main paper was omitted; dataset-specific tables and hyperparameter ranges available in the appendix were omitted as reference configurations.

References

  1. 1.State of Data Science and Machine Learning 2021. https://www.kaggle.com/kaggle-survey-2021.
  2. 2.Jannik Kossen, Neil Band, Clare Lyle, Aidan N. Gomez, Tom Rainforth, and Yarin Gal. Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning. arXiv:2106.02584 [cs, stat], June 2021.
  3. 3.Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting Deep Learning Models for Tabular Data. arXiv:2106.11959 [cs], June 2021.
  4. 4.Ravid Shwartz-Ziv and Amitai Armon. Tabular Data: Deep Learning is Not All You Need. arXiv:2106.03253 [cs], June 2021.
  5. 5.ImageNet: A large-scale hierarchical image database | IEEE Conference Publication | IEEE Xplore. https://ieeexplore.ieee.org/document/5206848.
  6. 6.Zachary C. Lipton and Jacob Steinhardt. Troubling Trends in Machine Learning Scholarship: Some ML papers suffer from flaws that could mislead the public and stymie future research. Queue, 17 (1):Pages 80:45–Pages 80:77, February 2019. ISSN 1542-7730. doi: 10.1145/3317287.3328534.
  7. 7.Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Tal Arbel, Chris Pal, Gael Varoquaux, and Pascal Vincent. Accounting for Variance in Machine Learning Benchmarks. Proceedings of Machine Learning and Systems, 3:747–769, March 2021.
  8. 8.Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. OpenML: Networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):49–60, June 2014. ISSN 1931-0145, 1931-0153. doi: 10.1145/2641190.2641198.
  9. 9.Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep Neural Networks and Tabular Data: A Survey. arXiv:2110.01889 [cs], October 2021.
  10. 10.John T. Hancock and Taghi M. Khoshgoftaar. Survey on categorical data for neural networks. Journal of Big Data, 7(1):28, April 2020. ISSN 2196-1115. doi: 10.1186/s40537-020-00305-w.
  11. 11.Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. VIME: Extending the Success of Self- and Semi-supervised Learning to Tabular Domain. In Advances in Neural Information Processing Systems, volume 33, pages 11033–11043. Curran Associates, Inc., 2020.
  12. 12.Nathan Lay, Adam P. Harrison, Sharon Schreiber, Gitesh Dawer, and Adrian Barbu. Random Hinge Forest for Differentiable Learning. arXiv:1802.03882 [cs, stat], March 2018.
  13. 13.Sergei Popov, S. Morozov, and Artem Babenko. Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data. undefined, 2020.
  14. 14.Ami Abutbul, Gal Elidan, Liran Katzir, and Ran El-Yaniv. DNF-Net: A Neural Architecture for Tabular Data, June 2020.
  15. 15.Thomas M. Hehn, Julian F. P. Kooij, and F. Hamprecht. End-to-End Learning of Decision Trees and Forests. undefined, 2019.
  16. 16.Ryutaro Tanno, Kai Arulkumaran, D. Alexander, A. Criminisi, and A. Nori. Adaptive Neural Trees. undefined, 2019.
  17. 17.Y. Chen. Attention augmented differentiable forest for tabular data. undefined, 2020.
  18. 18.Peter Kontschieder, Madalina Fiterau, Antonio Criminisi, and Samuel Rota Bulo. Deep Neural Decision Forests. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1467–1475, Santiago, Chile, December 2015. IEEE. ISBN 978-1-4673-8391-2. doi: 10.1109/ICCV.2015.172.
  19. 19.I. D. Rodriguez, Taylor W. Killian, Sung-Hyun Son, and M. Gombolay. Interpretable Reinforcement Learning via Differentiable Decision Trees. undefined, 2019.
  20. 20.Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. DeepFM: A Factorization-Machine based Neural Network for CTR Prediction, March 2017.
  21. 21.Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom Goldstein. SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training. arXiv:2106.01342 [cs, stat], June 2021.
  22. 22.Sercan Ö Arik and Tomas Pfister. TabNet: Attentive Interpretable Tabular Learning. undefined, 2019.
  23. 23.Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. TabTransformer: Tabular Data Modeling Using Contextual Embeddings. arXiv:2012.06678 [cs], December 2020.
  24. 24.Karim Lounici, Katia Meziani, and Benjamin Riu. Muddling Label Regularization: Deep Learning for Tabular Datasets. arXiv:2106.04462 [cs], June 2021.
  25. 25.Ira Shavitt and Eran Segal. Regularization Learning Networks: Deep Learning for Tabular Datasets. arXiv:1805.06440 [cs, stat], October 2018.
  26. 26.Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned Simple Nets Excel on Tabular Datasets, November 2021a.
  27. 27.James Fiedler. Simple Modifications to Improve Tabular Neural Networks. arXiv:2108.03214 [cs], August 2021.
  28. 28.Rich Caruana and Alexandru Niculescu-Mizil. An empirical comparison of supervised learning algorithms. In Proceedings of the 23rd International Conference on Machine Learning - ICML ’06, pages 161–168, Pittsburgh, Pennsylvania, 2006. ACM Press. ISBN 978-1-59593-383-6. doi: 10.1145/1143844.1143865.
  29. 29.Manuel Fernández-Delgado, Eva Cernadas, Senén Barro, and Dinani Amorim. Do we Need Hundreds of Classifiers to Solve Real World Classification Problems? Journal of Machine Learning Research, 15(90):3133–3181, 2014. ISSN 1533-7928.
  30. 30.Sherif Sakr, Radwa Elshawi, Amjad M. Ahmed, Waqas T. Qureshi, Clinton A. Brawner, Steven J. Keteyian, Michael J. Blaha, and Mouaz H. Al-Mallah. Comparison of machine learning techniques to predict all-cause mortality using fitness data: The Henry ford exercIse testing (FIT) project. BMC Medical Informatics and Decision Making, 17(1):174, December 2017. ISSN 1472-6947. doi: 10.1186/s12911-017-0566-6.
  31. 31.Alexandru Korotcov, Valery Tkachenko, Daniel P. Russo, and Sean Ekins. Comparison of Deep Learning With Multiple Machine Learning Methods and Metrics Using Diverse Drug Discovery Data Sets. https://pubs.acs.org/doi/pdf/10.1021/acs.molpharmaceut.7b00578, November 2017.
  32. 32.Shahadat Uddin, Arif Khan, Md Ekramul Hossain, and Mohammad Ali Moni. Comparing different supervised machine learning algorithms for disease prediction. BMC Medical Informatics and Decision Making, 19(1):281, December 2019. ISSN 1472-6947. doi: 10.1186/s12911-019-1004-8.
  33. 33.Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep Neural Networks and Tabular Data: A Survey. arXiv:2110.01889 [cs], February 2022.
  34. 34.Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems, February 2020.
  35. 35.Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael G. Mantovani, Jan N. van Rijn, and Joaquin Vanschoren. OpenML Benchmarking Suites, November 2021.
  36. 36.Pieter Gijsbers, Erin LeDell, Janek Thomas, Sébastien Poirier, Bernd Bischl, and Joaquin Vanschoren. An Open Source AutoML Benchmark, July 2019.
  37. 37.Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-Normalizing Neural Networks. arXiv:1706.02515 [cs, stat], September 2017.
  38. 38.Alexandre Perez-Lebel, Gaël Varoquaux, Marine Le Morvan, Julie Josse, and Jean-Baptiste Poline. Benchmarking missing-values approaches for predictive models on health databases. GigaScience, 11:giac013, January 2022. ISSN 2047-217X. doi: 10.1093/gigascience/giac013.
  39. 39.J Bergstra, D Yamins, and D.D Cox. Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures. 2013.
  40. 40.Brent Komer, James Bergstra, and Chris Eliasmith. Hyperopt-Sklearn: Automatic Hyperparameter Configuration for Scikit-Learn. In Python in Science Conference, pages 32–37, Austin, Texas, 2014. doi: 10.25080/Majora-14bd3278-006.
  41. 41.Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter. Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning, September 2021.
  42. 42.Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme. Learning hyperparameter optimization initializations. In 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA), pages 1–10, October 2015. doi: 10.1109/DSAA.2015.7344817.
  43. 43.Tianqi Chen and Carlos Guestrin. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794, August 2016. doi: 10.1145/2939672.2939785.
  44. 44.Jannik Kossen, Neil Band, Clare Lyle, Aidan N. Gomez, Tom Rainforth, and Yarin Gal. Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning, February 2022.
  45. 45.Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron Courville. On the Spectral Bias of Neural Networks, May 2019.
  46. 46.Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Regularization is all you Need: Simple Neural Nets can Excel on Tabular Data. arXiv:2106.11189 [cs], June 2021b.
  47. 47.Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geoffrey Hinton. Neural Additive Models: Interpretable Machine Learning with Neural Nets, October 2021.
  48. 48.Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On Embeddings for Numerical Features in Tabular Deep Learning. arXiv:2203.05556 [cs], March 2022.
  49. 49.Andrew Y. Ng. Feature selection, L 1 vs. L 2 regularization, and rotational invariance. In Twenty-First International Conference on Machine Learning - ICML ’04, page 78, Banff, Alberta, Canada, 2004. ACM Press. doi: 10.1145/1015330.1015435.
  50. 50.Patricio Cerda, Gaël Varoquaux, and Balázs Kégl. Similarity encoding for learning with dirty categorical variables. Machine Learning, 107(8):1477–1494, 2018.
  51. 51.Patricio Cerda and Gaël Varoquaux. Encoding high-cardinality string categorical variables. IEEE Transactions on Knowledge and Data Engineering, 2020.
  52. 52.Thais Mayumi Oshiro, Pedro Santoro Perez, and José Augusto Baranauskas. How Many Trees in a Random Forest? In Petra Perner, editor, Machine Learning and Data Mining in Pattern Recognition, Lecture Notes in Computer Science, pages 154–168, Berlin, Heidelberg, 2012. Springer. ISBN 978-3-642-31537-4. doi: 10.1007/978-3-642-31537-4_13.
  53. 53.Lukas Biewald. Experiment Tracking with Weights and Biases, 2020.
  54. 54.Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, ˙Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261–272, 2020. doi: 10.1038/s41592-019-0686-2.

Citation

MLA
Grinsztajn, L., et al. “Why Do Tree-based Models Still Outperform Deep Learning on Tabular Data?”. arXiv, 2022, http://arxiv.org/abs/2207.08815v1.
APA
Grinsztajn, L., Oyallon, E., & Varoquaux, G. (2022). Why do tree-based models still outperform deep learning on tabular data?. arXiv. http://arxiv.org/abs/2207.08815v1
Chicago
Grinsztajn, L., E. Oyallon, and G. Varoquaux. 2022. “Why Do Tree-based Models Still Outperform Deep Learning on Tabular Data?”. arXiv. http://arxiv.org/abs/2207.08815v1.
Harvard
Grinsztajn, L., Oyallon, E. and Varoquaux, G. (2022) “Why do tree-based models still outperform deep learning on tabular data?”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2207.08815v1.
Vancouver
1. Grinsztajn L, Oyallon E, Varoquaux G (2022) Why do tree-based models still outperform deep learning on tabular data?. arXiv

BibTeX

@article{grinsztajn2022why,
  title = {Why do tree-based models still outperform deep learning on tabular data?},
  author = {Grinsztajn, Léo and Oyallon, Edouard and Varoquaux, Gaël},
  year = {2022},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2207.08815v1},
  eprint = {2207.08815}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors