Back

Tabular Data: Deep Learning is Not All You Need

Ravid Shwartz-ZivAmitai Armon

article2021Information Fusion2,296 citations

Demonstrates that XGBoost consistently outperforms specialized deep learning architectures across tabular benchmarks while requiring far less tuning, establishing that deep models are most valuable when ensembled with tree-based methods rather than used as standalone replacements.

deep learningensemble classifiersensemble modelsDeepModelsTabular Data
arXiv: 2106.03253
  • Paper: TabNet: Attentive Interpretable Tabular Learning, Sercan Ö. Arik et al. (2019). Introduces TabNet, one of the primary deep learning architectures for tabular data whose empirical claims of outperforming gradient-boosted trees are directly evaluated and challenged by the source paper.
  • Paper: XGBoost: A Scalable Tree Boosting System, Tianqi Chen et al. (2016). Presents XGBoost, the foundational scalable gradient tree boosting framework that serves as the central baseline against which deep tabular architectures are benchmarked.
  • Paper: A comparative analysis of gradient boosting algorithms, Candice Bentéjac et al. (2019). Provides a comprehensive comparative analysis of modern gradient boosting algorithms, establishing standard benchmark methodologies and hyperparameter tuning dynamics for tree ensembles on tabular datasets.
  • Paper: CatBoost: unbiased boosting with categorical features, Liudmila Prokhorenkova et al. (2018). Introduces CatBoost and key techniques for handling categorical features in tree boosting, offering vital context for why tree ensembles excel on tabular data.
  • Paper: Greedy function approximation: A gradient boosting machine, Jerome H. Friedman (2001). Establishes the fundamental theoretical framework of gradient boosting machines that underpins tree ensemble baselines evaluated in the source study.
  • Paper: Accurate predictions on small data with a tabular foundation model, Noah Hollmann et al. (2025). Advances beyond the limitations of earlier deep tabular models discussed in the source by introducing TabPFN, a transformer-based tabular foundation model that achieves competitive predictive performance with in-context learning without extensive tuning.
Cover for Tabular Data: Deep Learning is Not All You Need

Abstract

A key element in solving real-life data science problems is selecting the types of models to use. Tree ensemble models (such as XGBoost) are usually recommended for classification and regression problems with tabular data. However, several deep learning models for tabular data have recently been proposed, claiming to outperform XGBoost for some use cases. This paper explores whether these deep models should be a recommended option for tabular data by rigorously comparing the new deep models to XGBoost on various datasets. In addition to systematically comparing their performance, we consider the tuning and computation they require. Our study shows that XGBoost outperforms these deep models across the datasets, including the datasets used in the papers that proposed the deep models. We also demonstrate that XGBoost requires much less tuning. On the positive side, we show that an ensemble of deep models and XGBoost performs better on these datasets than XGBoost alone.

Topics

deep learningensemble classifiersensemble modelsrandom forestssocial sciencestuning method

Keywords

DeepModelsTabular DataLearningDatasetsDeep LearningDeep ModelsXGBoostModelEnsemble

Table of Contents

  • 1 Introduction
  • 2 Background
  • 2.1 Deep Neural Models for Tabular Data
  • 2.2 Model Ensemble
  • 3 Comparing the Models
  • 3.1 Experimental Setup
  • 3.1.1 Data-sets Description
  • 3.1.2 Implementation Details
  • 3.2 Results
  • 4 Discussion and Conclusions
  • References
  • A Tabular Data-sets Description
  • B Optimization of hyperparameters
  • B.1 CATBOOST
  • B.2 XGBoost
  • B.3 NODE
  • B.4 TabNet
  • B.5 DNF-Net
  • B.6 1D-CNN

Knowls

  1. Knowl 1 — Performance of Deep Tabular Architectures vs. XGBoost on Tabular Benchmarks

    data/table

    Across eleven tabular benchmarks spanning classification and regression tasks, deep learning models (TabNet, NODE, DNF-Net, and 1D-CNN) were evaluated alongside XGBoost, simple uniform ensembles, deep-only ensembles, and combined deep-plus-tree ensembles. Performance is measured using Mean Squared Error (MSE) for regression tasks (Rossmann and YearPrediction) and Cross-Entropy loss multiplied by a factor of 100 for classification tasks. Values represent the mean and standard error of the mean (SEM) over four training runs.

    Model Rossmann CoverType Higgs Gas Eye Gesture
    XGBoost 490.18±1.19490.18 \pm 1.19 3.13±0.093.13 \pm 0.09 21.62±0.3321.62 \pm 0.33 2.18±0.202.18 \pm 0.20 56.07±0.6556.07 \pm 0.65 80.64±0.8080.64 \pm 0.80
    NODE 488.59±1.24488.59 \pm 1.24 4.15±0.134.15 \pm 0.13 21.19±0.6921.19 \pm 0.69 2.17±0.182.17 \pm 0.18 68.35±0.6668.35 \pm 0.66 92.12±0.8292.12 \pm 0.82
    DNF-Net 503.83±1.41503.83 \pm 1.41 3.96±0.113.96 \pm 0.11 23.68±0.8323.68 \pm 0.83 1.44±0.091.44 \pm 0.09 68.38±0.6568.38 \pm 0.65 86.98±0.7486.98 \pm 0.74
    TabNet 485.12±1.93485.12 \pm 1.93 3.01±0.083.01 \pm 0.08 21.14±0.2021.14 \pm 0.20 1.92±0.141.92 \pm 0.14 67.13±0.6967.13 \pm 0.69 96.42±0.8796.42 \pm 0.87
    1D-CNN 493.81±2.23493.81 \pm 2.23 3.51±0.133.51 \pm 0.13 22.33±0.7322.33 \pm 0.73 1.79±0.191.79 \pm 0.19 67.90±0.6467.90 \pm 0.64 97.89±0.8297.89 \pm 0.82
    Simple Ensemble 488.57±2.14488.57 \pm 2.14 3.19±0.183.19 \pm 0.18 22.46±0.3822.46 \pm 0.38 2.36±0.132.36 \pm 0.13 58.72±0.6758.72 \pm 0.67 89.45±0.8989.45 \pm 0.89
    Deep Ensemble w/o XGBoost 489.94±2.09489.94 \pm 2.09 3.52±0.103.52 \pm 0.10 22.41±0.5422.41 \pm 0.54 1.98±0.131.98 \pm 0.13 69.28±0.6269.28 \pm 0.62 93.50±0.7593.50 \pm 0.75
    Deep Ensemble w XGBoost 485.33±1.29485.33 \pm 1.29 2.99±0.082.99 \pm 0.08 22.34±0.8122.34 \pm 0.81 1.69±0.101.69 \pm 0.10 59.43±0.6059.43 \pm 0.60 78.93±0.7378.93 \pm 0.73
    Model YearPrediction MSLR Epsilon Shrutime Blastchar
    XGBoost 77.98±0.1177.98 \pm 0.11 55.43±0.0255.43 \pm 0.02 11.12±0.0311.12 \pm 0.03 13.82±0.1913.82 \pm 0.19 20.39±0.2120.39 \pm 0.21
    NODE 76.39±0.1376.39 \pm 0.13 55.72±0.0355.72 \pm 0.03 10.39±0.0110.39 \pm 0.01 14.61±0.1014.61 \pm 0.10 21.40±0.2521.40 \pm 0.25
    DNF-Net 81.21±0.1881.21 \pm 0.18 56.83±0.0356.83 \pm 0.03 12.23±0.0412.23 \pm 0.04 16.80±0.0916.80 \pm 0.09 27.91±0.1727.91 \pm 0.17
    TabNet 83.19±0.1983.19 \pm 0.19 56.04±0.0156.04 \pm 0.01 11.92±0.0311.92 \pm 0.03 14.94±0.1314.94 \pm 0.13 23.72±0.1923.72 \pm 0.19
    1D-CNN 78.94±0.1478.94 \pm 0.14 55.97±0.0455.97 \pm 0.04 11.08±0.0611.08 \pm 0.06 15.31±0.1615.31 \pm 0.16 24.68±0.2224.68 \pm 0.22
    Simple Ensemble 78.01±0.1778.01 \pm 0.17 55.46±0.0455.46 \pm 0.04 11.07±0.0411.07 \pm 0.04 13.61±0.1413.61 \pm 0.14 21.18±0.1721.18 \pm 0.17
    Deep Ensemble w/o XGBoost 78.99±0.1178.99 \pm 0.11 55.59±0.0355.59 \pm 0.03 10.95±0.0110.95 \pm 0.01 14.69±0.1114.69 \pm 0.11 24.25±0.2224.25 \pm 0.22
    Deep Ensemble w XGBoost 76.19±0.2176.19 \pm 0.21 55.38±0.0155.38 \pm 0.01 11.18±0.0111.18 \pm 0.01 13.10±0.1513.10 \pm 0.15 20.18±0.1620.18 \pm 0.16

    The benchmark shows two core patterns: first, each deep model demonstrates competitive performance primarily on the datasets introduced in its originating paper, but underperforms on datasets outside its original study. Second, XGBoost outperforms deep models on 8 of the 11 datasets when evaluated on datasets new to those models (p<0.005p < 0.005 under the non-parametric Friedman test). The combined ensemble of XGBoost and deep neural networks achieves the overall best results across 7 of 11 tasks.

  2. Knowl 2 — Generalization Deterioration of Deep Tabular Models on Unseen Datasets

    empirical result

    When evaluated on datasets not included in their original introducing papers (unseen datasets), deep neural models exhibit substantial performance degradation compared to the dataset-best model. The geometric mean of the relative performance increase/deterioration across unseen datasets is:

    Model Average Relative Deterioration (%)
    Deep Ensemble w XGBoost 2.32
    Simple Ensemble 3.15
    XGBoost 3.34
    Deep Ensemble w/o XGBoost 6.91
    1D-CNN 7.56
    TabNet 10.51
    DNF-Net 11.96
    NODE 14.21

    Individual deep tabular networks degrade by 7.56%7.56\% to 14.21%14.21\% relative to the top performer on unseen datasets, whereas standalone XGBoost experiences only a 3.34%3.34\% deterioration. The hybrid ensemble combining all deep architectures with XGBoost attains the lowest deterioration (2.32%2.32\%, with all pairwise differences yielding p<0.005p < 0.005 except one).

  3. Knowl 3 — Ensembling Deep Learning Models with XGBoost for Tabular Data

    model/method

    To combine deep models (TabNet, NODE, DNF-Net, 1D-CNN) and gradient boosted decision trees (XGBoost), two ensembling schemes are used over KK base models with parameter sets θ1,…,θK\theta_1, \dots, \theta_K:

    1. Uniform Mixture Ensemble: p(y∣x)=1K∑k=1Kp(y∣x,θk)p(y \mid x) = \frac{1}{K} \sum_{k=1}^K p(y \mid x, \theta_k)

    2. Validation-Loss-Weighted Ensemble: p(y∣x)=∑k=1Klkvalp(y∣x,θk)p(y \mid x) = \sum_{k=1}^K l_k^{\text{val}} p(y \mid x, \theta_k) where lkvall_k^{\text{val}} represents the normalized validation loss weight assigned to the kk-th model such that ∑k=1Klkval=1\sum_{k=1}^K l_k^{\text{val}} = 1. Each base learner is trained independently on the entire training set to minimize bias, and predictions are aggregated to reduce prediction variance.

  4. Knowl 4 — Comparative Superiority of Hybrid Deep-Tree Ensembles over Homogeneous Ensembles

    empirical result

    Ensembling both XGBoost and deep tabular networks yields statistically significant improvements over homogeneous ensembles:

    1. A combined ensemble of XGBoost and deep models outperforms an ensemble consisting solely of deep networks (Deep Ensemble w/o XGBoost) across all tested datasets (p<0.005p < 0.005).
    2. A combined ensemble of XGBoost and deep models outperforms an ensemble composed exclusively of classical machine learning models (XGBoost, SVM, and CatBoost) with p<0.005p < 0.005.

    This confirms that deep architectures and tree-based gradient boosting models learn complementary representations on tabular datasets, making heterogeneous ensembling strictly superior to ensembling either model family alone.

  5. Knowl 5 — Validation-Loss-Based Model Selection for Ensemble Pruning

    empirical result

    When computational or latency constraints restrict the number of base models in an ensemble, greedy selection based on validation loss provides the most effective subset selection strategy.

    Evaluating three model selection policies—(1) ranking models by validation loss, (2) selecting models based on sample-level predictive confidence/uncertainty, and (3) random ordering—demonstrates that validation-loss-based selection achieves near-optimal ensemble loss with only the top 3 models. In contrast, confidence-based and random selection strategies exhibit slower convergence and significantly inferior predictive performance for early subset sizes (p<0.005p < 0.005).

  6. Knowl 6 — Hyperparameter Optimization Efficiency and Convergence Dynamics

    empirical result

    Under Bayesian hyperparameter optimization using HyperOpt across 1,000 iterations:

    1. Iteration Convergence: XGBoost reaches a performance plateau in significantly fewer optimization steps than deep tabular architectures (NODE, DNF-Net, TabNet, 1D-CNN), which require hundreds of additional trials to discover competitive configurations.
    2. Computational Runtime: Standalone XGBoost training and tuning is more than an order of magnitude faster in wall-clock time compared to the deep tabular neural networks.
    3. Default Robustness: The rapid plateau of XGBoost indicates greater robustness to initial hyperparameter settings compared to tabular deep networks, whose performance is highly sensitive to architecture-specific hyperparameter search spaces.

  7. Knowl 7 — Tabular Evaluation Benchmark Suite and Standardization Protocol

    experimental setup

    The benchmark protocol encompasses 11 diverse tabular datasets spanning binary classification, multi-class classification, and regression tasks:

    Dataset Features Classes Samples Source / Paper
    Gesture Phase 32 5 9.8k9.8\text{k} OpenML (DNF-Net)
    Gas Concentrations 129 6 13.9k13.9\text{k} OpenML (DNF-Net)
    Eye Movements 26 3 10.9k10.9\text{k} OpenML (DNF-Net)
    Epsilon 2000 2 500k500\text{k} PASCAL Challenge 2008 (NODE)
    YearPrediction 90 1 515k515\text{k} Million Song Dataset (NODE)
    Microsoft (MSLR) 136 5 964k964\text{k} MSLR-WEB10K (NODE)
    Rossmann Store Sales 10 1 1018k1018\text{k} Kaggle (TabNet)
    Forest Cover Type 54 7 580k580\text{k} Kaggle (TabNet)
    Higgs Boson 30 2 800k800\text{k} Kaggle (TabNet)
    Shrutime 11 2 10k10\text{k} Kaggle (New dataset)
    Blastchar 20 2 7k7\text{k} Kaggle (New dataset)

    Standardization and Training Protocol:

    • Features are standardized to zero mean and unit variance using statistics derived strictly from training partitions.
    • Stratified splits follow the source papers (e.g., 80%/20%80\%/20\% train/validation for Epsilon, YearPrediction, MSLR, Shrutime, Blastchar; 70%/10%/20%70\%/10\%/20\% train/val/test for Eye, Gesture, Year).
    • Deep models are trained with the Adam optimizer without learning rate schedules, utilizing batch size optimization and early stopping after 100 consecutive epochs without validation loss improvement.
    • Hyperparameter optimization is executed for 1,000 steps per dataset per model using HyperOpt.

  8. Knowl 8 — Hyperparameter Search Spaces for Tree-Based and Deep Tabular Models

    experimental setup

    Hyperparameter search spaces optimized via Bayesian optimization (HyperOpt) for 1,000 steps across models are defined as follows:

    XGBoost:

    • Number of estimators: Uniform [100,4000][100, 4000]
    • Eta (learning rate): Log-Uniform [e−7,1][e^{-7}, 1]
    • Max depth: Discrete uniform [1,10][1, 10]
    • Subsample: Uniform [0.2,1][0.2, 1]
    • Colsample bytree: Uniform [0.2,1][0.2, 1]
    • Colsample bylevel: Uniform [0.2,1][0.2, 1]
    • Min child weight: Log-Uniform [e−16,e5][e^{-16}, e^5]
    • Alpha, Lambda, Gamma: Choice {0,Log-Uniform[e−16,e2]}\{0, \text{Log-Uniform}[e^{-16}, e^2]\}

    CatBoost:

    • Learning rate: Log-Uniform [e−5,1][e^{-5}, 1]
    • Random strength: Discrete uniform [1,20][1, 20]
    • Max size: Discrete uniform [0,25][0, 25]
    • L2 leaf regularization: Log-Uniform [1,10][1, 10]
    • Bagging temperature: Uniform [0,1][0, 1]
    • Leaf estimation iterations: Discrete uniform [1,20][1, 20]

    NODE:

    • Learning rate: Log-Uniform [e−5,1][e^{-5}, 1] and Log-Uniform [e−4,0.5][e^{-4}, 0.5]
    • Number of layers: Discrete uniform [1,10][1, 10]
    • Total tree count: Choice {256,512,1024,2048}\{256, 512, 1024, 2048\}
    • Tree depth: Discrete uniform [4,9][4, 9]
    • Tree output dimension: Discrete uniform [1,5][1, 5]
    • Batch size: Choice {512,1024,2048,4096,8192}\{512, 1024, 2048, 4096, 8192\}

    TabNet:

    • Learning rate: Log-Uniform [e−5,1][e^{-5}, 1]
    • Feature dimension, Output dimension: Discrete uniform [20,60][20, 60]
    • Number of steps: Discrete uniform [1,8][1, 8]
    • Batch normalization epsilon: Uniform [e−5,e−1][e^{-5}, e^{-1}]
    • Relaxation factor: Uniform [0.3,2][0.3, 2]
    • Batch size: Choice {512,1024,2048,4096,8192}\{512, 1024, 2048, 4096, 8192\}

    DNF-Net:

    • Number of formulas: Discrete uniform [256,2048][256, 2048]
    • Feature selection beta: Discrete uniform [10−2,2][10^{-2}, 2]
    • Learning rate: Log-Uniform [e−4,0.5][e^{-4}, 0.5]
    • Batch size: Choice {512,1024,2048,4096,8192}\{512, 1024, 2048, 4096, 8192\}

    1D-CNN:

    • Hidden layer sizes: Discrete uniform [100,4000][100, 4000]
    • Number of layers: Discrete uniform [1,6][1, 6]
    • Learning rate: Log-Uniform [e−4,0.5][e^{-4}, 0.5]
    • Batch size: Choice {512,1024,2048,4096,8192}\{512, 1024, 2048, 4096, 8192\}

Coverage note — All primary empirical findings, benchmark datasets, ensembling methods, subset selection dynamics, and hyperparameter search spaces have been fully captured. Summary overviews of prior published model architectures (TabNet, NODE, DNF-Net) were omitted as they represent existing work reviewed by the paper rather than original contributions.

References

  1. 1.J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
  2. 2.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  3. 3.Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A Generative Model for Raw Audio. In Proc. 9th ISCA Workshop on Speech Synthesis Workshop (SSW 9), page 125, 2016.
  4. 4.Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785–794, 2016.
  5. 5.Ravid Shwartz-Ziv, Amichai Painsky, and Naftali Tishby. Representation compression and generalization in deep neural networks. 2018.
  6. 6.Tomaso Poggio, Andrzej Banburski, and Qianli Liao. Theoretical issues in deep networks. Proceedings of the National Academy of Sciences, 117(48):30039–30045, 2020. ISSN 0027-8424. doi:10.1073/pnas.1907369117. URL https://www.pnas.org/content/117/48/30039.
  7. 7.Zoe Piran, Ravid Shwartz-Ziv, and Naftali Tishby. The dual information bottleneck. arXiv preprint arXiv:2006.04641, 2020.
  8. 8.Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810, 2017.
  9. 9.David H Wolpert and William G Macready. No free lunch theorems for optimization. IEEE transactions on evolutionary computation, 1(1):67–82, 1997.
  10. 10.Jerome H Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics, pages 1189–1232, 2001.
  11. 11.Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018a.
  12. 12.Sercan Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(8):6679–6687, May 2021. URL https://ojs.aaai.org/index.php/AAAI/article/view/16826.
  13. 13.Liran Katzir, Gal Elidan, and Ran El-Yaniv. Net-dnf: Effective deep modeling of tabular data. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id=73WTGs96kho.
  14. 14.Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreview.net/forum?id=r1eiu2VtwH.
  15. 15.Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  16. 16.Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi:10.18653/v1/W18-5446. URL https://aclanthology.org/W18-5446.
  17. 17.Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/6449f44a102fde848669bdd9eb6b76fa-Paper.pdf.
  18. 18.Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: Unbiased boosting with categorical features. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, page 6639–6649, Red Hook, NY, USA, 2018b. Curran Associates Inc.
  19. 19.Yun Zhao, Girija Chetty, and Dat Tran. Deep learning with xgboost for real estate appraisal. In 2019 IEEE Symposium Series on Computational Intelligence (SSCI), pages 1396–1401. IEEE, 2019.
  20. 20.S Ramraj, Nishant Uzir, R Sunil, and Shatadeep Banerjee. Experimenting xgboost algorithm for prediction and classification of different datasets. International Journal of Control Theory and Applications, 9:651–662, 2016.
  21. 21.Sarkhan Badirli, Xuanqing Liu, Zhengming Xing, Avradeep Bhowmik, Khoa Doan, and Sathiya S Keerthi. Gradient boosting neural networks: Grownet. arXiv preprint arXiv:2002.07971, 2020.
  22. 22.Hussein Hazimeh, Natalia Ponomareva, Petros Mol, Zhenyu Tan, and Rahul Mazumder. The tree ensemble layer: Differentiability meets conditional computation. In International Conference on Machine Learning, pages 4138–4148. PMLR, 2020.
  23. 23.Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678, 2020.
  24. 24.Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In Proceedings of the 31st international conference on neural information processing systems, pages 972–981, 2017.
  25. 25.Baosenguo. baosenguo/kaggle-moa-2nd-place-solution, 2021. URL https://github.com/baosenguo/Kaggle-MoA-2nd-Place-Solution.
  26. 26.Peter Kontschieder, Madalina Fiterau, Antonio Criminisi, and Samuel Rota Bulò. Deep neural decision forests. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1467–1475, 2015. doi:10.1109/ICCV.2015.172.
  27. 27.Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342, 2021.
  28. 28.Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Regularization is all you need: Simple neural nets can excel on tabular data. arXiv preprint arXiv:2106.11189, 2021.
  29. 29.Ira Shavitt and Eran Segal. Regularization learning networks: deep learning for tabular datasets. 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018.
  30. 30.Alex Beutel, Paul Covington, Sagar Jain, Can Xu, Jia Li, Vince Gatto, and Ed H. Chi. Latent cross: Making use of context in recurrent recommender systems. In WSDM 2018: The Eleventh ACM International Conference on Web Search and Data Mining, 2018.
  31. 31.Rich Caruana, Alexandru Niculescu-Mizil, Geoff Crew, and Alex Ksikes. Ensemble selection from libraries of models. In Proceedings of the twenty-first international conference on Machine learning, page 18, 2004.
  32. 32.Leo Breiman. Random forests. Machine learning, 45(1):5–32, 2001.
  33. 33.Leo Breiman. Bagging predictors. Machine learning, 24(2):123–140, 1996.
  34. 34.Yoav Freund, Robert E Schapire, et al. Experiments with a new boosting algorithm. In icml, volume 96, pages 148–156. Citeseer, 1996.
  35. 35.David H Wolpert. Stacked generalization. Neural networks, 5(2):241–259, 1992.
  36. 36.Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20(3):273–297, 1995.
  37. 37.Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://archive.ics.uci.edu/ml.
  38. 38.Kaggle. Rossmann store sales, 2019a. URL https://www.kaggle.com/c/rossmann-store-sales.
  39. 39.Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo. Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):49–60, 2014.
  40. 40.Tao Qin and Tie-Yan Liu. Introducing LETOR 4.0 datasets. CoRR, abs/1306.2597, 2013. URL http://arxiv.org/abs/1306.2597.
  41. 41.Pascal. Pascal large scale learning challenge, 2008. URL https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary.html.
  42. 42.Kaggle. Churn modelling, 2019b. URL https://www.kaggle.com/shrutimechlearn/churn-modelling.
  43. 43.IBM. Telco customer churn, 2019. URL https://community.ibm.com/community/user/businessanalytics/blogs/steven-macko/2019/07/11/telco-customer-churn-1113.
  44. 44.James Bergstra, Brent Komer, Chris Eliasmith, Dan Yamins, and David D Cox. Hyperopt: a python library for model selection and hyperparameter optimization. Computational Science & Discovery, 8(1):014008, 2015.
  45. 45.Milton Friedman. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, 32(200):675–701, 1937. doi:10.1080/01621459.1937.10503522. URL https://www.tandfonline.com/doi/abs/10.1080/01621459.1937.10503522.
  46. 46.Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 8th International Conference on Learning Representations, ICLR 2020, 2015.
  47. 47.Raphael Tang, Weijie Wang, Zhucheng Tu, and Jimmy Lin. An experimental analysis of the power consumption of convolutional neural networks for keyword spotting. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5479–5483. IEEE, 2018.
  48. 48.Rory Mitchell, Andrey Adinets, Thejaswi Rao, and Eibe Frank. Xgboost: Scalable gpu accelerated learning. arXiv preprint arXiv:1806.11248, 2018.

Citation

MLA
Shwartz-Ziv, R., and A. Armon. “Tabular Data: Deep Learning Is Not All You Need”. arXiv, 2021, http://arxiv.org/abs/2106.03253v2.
APA
Shwartz-Ziv, R., & Armon, A. (2021). Tabular Data: Deep Learning is Not All You Need. arXiv. http://arxiv.org/abs/2106.03253v2
Chicago
Shwartz-Ziv, R., and A. Armon. 2021. “Tabular Data: Deep Learning Is Not All You Need”. arXiv. http://arxiv.org/abs/2106.03253v2.
Harvard
Shwartz-Ziv, R. and Armon, A. (2021) “Tabular Data: Deep Learning is Not All You Need”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2106.03253v2.
Vancouver
1. Shwartz-Ziv R, Armon A (2021) Tabular Data: Deep Learning is Not All You Need. arXiv

BibTeX

@article{shwartzziv2021tabular,
  title = {Tabular Data: Deep Learning is Not All You Need},
  author = {Shwartz-Ziv, Ravid and Armon, Amitai},
  year = {2021},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2106.03253v2},
  eprint = {2106.03253}
}
Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/