Back

An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants

E. BauerRon Kohavi

article1999Machine Learning2,830 citations

Demonstrates through empirical bias-variance decomposition how bagging and boosting alter classification error, showing that bagging primarily reduces variance in unstable models while boosting reduces both bias and variance but struggles with noisy data and stable learners like Naive-Bayes.

Listen

The article addresses the challenge of improving classification accuracy using voting methods such as Bagging and boosting algorithms like AdaBoost. These techniques have demonstrated success on both artificial and real-world datasets, yet the reasons they reduce error for certain inducers remain unclear, particularly regarding their impact on bias and variance. Understanding these mechanisms matters because accurate classifiers affect practical decisions in many domains.

The article set out to evaluate Bagging, AdaBoost, Arc-x4, and several variants in conjunction with decision tree inducers and a Naive-Bayes inducer. It aimed to determine why and when these perturbation, reweighting, and combination methods lower classification error.

The authors conducted a large empirical study across 14 datasets, each with at least 1,000 instances. They applied bias-variance decomposition to isolate effects, tested variants including pruning choices, probabilistic estimates, and backfitting, and performed sanity checks against prior experiments. Training and test splits followed learning-curve guidance to ensure room for improvement.

Bagging reduced variance for unstable inducers like decision trees, cutting average error from 12.6 percent to 10.4 percent, while boosting methods lowered both bias and variance for the same inducers, achieving a 27 percent relative error reduction. Boosting increased variance for the stable Naive-Bayes inducer yet still improved overall accuracy. Arc-x4 performed comparably to AdaBoost when resampling was used but worse with reweighting alone. No-pruning combined with probabilistic estimates further improved Bagging, and mean-squared error dropped substantially across voting methods. Boosting sometimes raised error on noisy datasets.

These findings indicate that voting methods deliver meaningful gains in accuracy and calibration where comprehensibility is secondary. They suggest practitioners should prefer boosting or refined Bagging variants over single trees or Naive-Bayes for many tasks, though gains vary by data noise and inducer stability.

The article recommends exploring noise-robust boosting variants, parallel implementations, and methods to preserve interpretability. Additional work is needed on tree-size dynamics and handling zero-error trials during boosting.

The study is limited to two inducer families and 14 datasets; results may not generalize to other algorithms or noisier domains. Confidence is moderate given the controlled experimental design, yet caution is warranted for noisy data or when many trials are required.

Blue Martini SoftwareStanford Universityboostingclassification algorithmclassification modelsBaggingAdaBoostArc-x4
Bauer et al (1999).pdf
Cover for An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants

Abstract

Methods for voting classification algorithms, such as Bagging and AdaBoost, have been shown to be very successful in improving the accuracy of certain classifiers for artificial and real-world datasets. We review these algorithms and describe a large empirical study comparing several variants in conjunction with a decision tree inducer (three variants) and a Naive-Bayes inducer. The purpose of the study is to improve our understanding of why and when these algorithms, which use perturbation, reweighting, and combination techniques, affect classification error. We provide a bias and variance decomposition of the error to show how different methods and variants influence these two terms. This allowed us to determine that Bagging reduced variance of unstable methods, while boosting methods (AdaBoost and Arc-x4) reduced both the bias and variance of unstable methods but increased the variance for Naive-Bayes, which was very stable. We observed that Arc-x4 behaves differently than AdaBoost if reweighting is used instead of resampling, indicating a fundamental difference. Voting variants, some of which are introduced in this paper, include: pruning versus no pruning, use of probabilistic estimates, weight perturbations (Wagging), and backfitting of data. We found that Bagging improves when probabilistic estimates in conjunction with no-pruning are used, as well as when the data was backfit. We measure tree sizes and show an interesting positive correlation between the increase in the average tree size in AdaBoost trials and its success in reducing the error. We compare the mean-squared error of voting methods to non-voting methods and show that the voting methods lead to large and significant reductions in the mean-squared errors. Practical problems that arise in implementing boosting algorithms are explored, including numerical instabilities and underflows. We use scatterplots that graphically show how AdaBoost reweights instances, emphasizing not only “hard” areas but also outliers and noise.

Topics

boostingclassification algorithmclassification modelsnaive bayesmean square errorsvmensemble learningensemble classifiersrandom forests

Keywords

BaggingAdaBoostArc-x4Decision TreeNaive BayesBias Variance DecompositionProbabilistic EstimateMean Squared ErrorWaggingBackfitting

Table of Contents

  • 1. Introduction
  • 2. Notation
  • 3. The base inducers
  • 3.1. The decision tree inducers
  • 3.2. The Naive-Bayes Inducer
  • 4. The voting algorithms
  • 4.1. The Bagging algorithm
  • 4.2. Boosting
  • 4.3. Arc-x4
  • 5. The bias and variance decomposition
  • 6. Experimental design
  • 6.1. Desiderata for comparisons
  • 6.2. *Sanity check for correctness*
  • 6.3. *Runs and measurements*
  • 7.2. Pruning
  • 7.3. *Using probabilistic estimates*
  • 7.4. *Mean-squared errors*
  • 7.5. *Wagging and backfitting data*
  • 7.6. Conclusions on Bagging
  • 8. Boosting algorithms: AdaBoost and Arc-x4
  • 8.1. Numerical instabilities and a detailed boosting example
  • 8.2. AdaBoost: Error, bias, and variance
  • 8.3. *Arc-x4: Error, bias, and variance*
  • 8.4. Conclusions for boosting
  • 9. Future work
  • 10. Conclusions
  • Acknowledgments
  • Note
  • References

Knowls

  1. Knowl 1 — Kohavi-Wolpert Bias-Variance Decomposition for Classification Error

    equation

    The bias plus variance decomposition for supervised classification under zero-one loss separates the expected classification error into intrinsic target noise, squared bias, and variance.

    Let x∈Xx \in X denote an input attribute vector, YY the finite set of discrete class labels, YFY_F the random variable representing the true target class label associated with xx, and YHY_H the random variable representing the class predicted for xx by an inducer trained on a randomly sampled training set of fixed size mm. The expected classification error rate over the input distribution P(x)P(x) is:

    Error=∑x∈XP(x)(σx2+biasx2+variancex)\text{Error} = \sum_{x \in X} P(x) \left( \sigma_x^2 + \text{bias}_x^2 + \text{variance}_x \right)

    where the individual point-wise components are defined as:

    σx2≡12(1−∑y∈YP(YF=y∣x)2)\sigma_x^2 \equiv \frac{1}{2} \left( 1 - \sum_{y \in Y} P(Y_F = y \mid x)^2 \right)

    biasx2≡12∑y∈Y[P(YF=y∣x)−P(YH=y∣x)]2\text{bias}_x^2 \equiv \frac{1}{2} \sum_{y \in Y} \left[ P(Y_F = y \mid x) - P(Y_H = y \mid x) \right]^2

    variancex≡12(1−∑y∈YP(YH=y∣x)2)\text{variance}_x \equiv \frac{1}{2} \left( 1 - \sum_{y \in Y} P(Y_H = y \mid x)^2 \right)

    Here, σx2\sigma_x^2 is the intrinsic target noise (the Bayes-optimal error lower bound), biasx2\text{bias}_x^2 measures how closely the learning algorithm's average prediction distribution matches the true posterior distribution of the target, and variancex\text{variance}_x measures the degree of fluctuation in the induced classifier across different training sets of size mm drawn from the instance distribution.

  2. Knowl 2 — Comparative Bias and Variance Dynamics of Bagging and Boosting

    empirical result

    Empirical evaluation across 14 benchmark datasets demonstrates distinct error-reduction mechanisms between Bagging and boosting algorithms (AdaBoost and Arc-x4) across different inducer stability regimes:

    1. Unstable Inducers (MC4 Decision Trees): MC4 baseline achieves an average error of 12.6% (bias 6.9%, variance 5.7%). Bagging MC4 reduces average error to 10.4% (a 14.5% relative reduction), which is driven almost entirely by variance reduction (variance drops from 5.7% to 3.5%, a 29% relative reduction) while bias remains essentially flat (6.8%). AdaBoost MC4 reduces average error to 9.8% (a 27% relative reduction) by simultaneously reducing both bias (from 6.9% to 5.7%, a 32% relative reduction) and variance (from 5.7% to 4.1%, a 16% relative reduction).

    2. Stable Inducers (Naive-Bayes): Naive-Bayes baseline achieves an average error of 13.6% (bias 10.8%, variance 2.8%). Bagging provides negligible improvement (13.2% error, variance 2.5%) because base variance is already low. AdaBoost reduces average error to 12.3% (a 24% relative reduction) by decreasing bias from 10.8% to 8.7%, but it increases variance from 2.8% to 3.6% due to instability introduced into the entropy-based continuous feature discretization across reweighted training distributions.

    3. Restricted Inducers (One-Level Discretized Decision Trees, MC4(1)-disc): MC4(1)-disc baseline achieves 33.0% error (bias 24.4%, variance 8.6%). Bagging reduces error to 31.5% (variance drops to 6.5%). AdaBoost reduces error to 27.1% (bias drops to 19.2%, variance to 8.0%). Arc-x4 with resampling outperforms AdaBoost on MC4(1)-disc, reducing error to 24.6% (bias drops to 17.4%, variance to 7.2%).

  3. Knowl 3 — AdaBoost.M1 with Direct Instance Reweighting

    algorithm

    AdaBoost.M1 trains an ensemble of TT classifiers sequentially on weighted versions of a training set S={(x1,y1),…,(xm,ym)}S = \{(x_1, y_1), \dots, (x_m, y_m)\}. Weights are updated based on classification accuracy, doubling the aggregate weight of misclassified instances and halving correctly classified instances without requiring explicit sum renormalization.

    Input: Training set S={(x1,y1),…,(xm,ym)}S = \{(x_1, y_1), \dots, (x_m, y_m)\}, Base inducer II, Number of trials TT
    Output: Combined classifier C∗C^*
    Initialize instance weights: wj=1w_j = 1 for all j∈{1,…,m}j \in \{1, \dots, m\}
    for i=1i = 1 to TT:
        S′=SS' = S with instance weights ww
        Ci=I(S′)C_i = I(S')
        ϵi=1m∑j:Ci(xj)≠yjwj\epsilon_i = \frac{1}{m} \sum_{j: C_i(x_j) \ne y_j} w_j
        if ϵi>0.5\epsilon_i > 0.5:
            Generate bootstrap sample from original SS with unit weights (up to 25 attempts)
            if successful:
                Re-induce CiC_i on bootstrap sample and recompute ϵi\epsilon_i
            else:
                Exit loop
        if ϵi=0\epsilon_i = 0:
            C∗(x)=Ci(x)C^*(x) = C_i(x)
            return C∗C^*
        βi=ϵi/(1−ϵi)\beta_i = \epsilon_i / (1 - \epsilon_i)
        for each j∈{1,…,m}j \in \{1, \dots, m\}:
            if Ci(xj)≠yjC_i(x_j) \ne y_j:
                wj=wj/(2ϵi)w_j = w_j / (2 \epsilon_i)
            else:
                wj=wj/(2(1−ϵi))w_j = w_j / (2 (1 - \epsilon_i))
            if wj<10−6w_j < 10^{-6}:
                wj=10−6w_j = 10^{-6}
    for any test instance xx:
        C∗(x)=arg⁡max⁡y∈Y∑i:Ci(x)=ylog⁡(1/βi)C^*(x) = \arg\max_{y \in Y} \sum_{i: C_i(x) = y} \log(1 / \beta_i)
    return C∗C^*

    Dividing incorrect instances by 2ϵi2\epsilon_i and correct instances by 2(1−ϵi)2(1-\epsilon_i) ensures that the sum of weights of incorrectly classified instances equals m/2m/2 and the sum of correctly classified instances equals m/2m/2, keeping the total dataset weight invariant at mm.

  4. Knowl 4 — Arc-x4 Algorithm and Resampling versus Reweighting Dynamics

    algorithm

    Arc-x4 (Adaptively Resample and Combine with a 4th-power polynomial) constructs an ensemble of TT classifiers where instance weights depend strictly on cumulative classification error counts rather than exponential loss.

    Input: Training set S={(x1,y1),…,(xm,ym)}S = \{(x_1, y_1), \dots, (x_m, y_m)\}, Base inducer II, Number of trials TT
    Output: Combined classifier C∗C^*
    Initialize error counts: e(xj)=0e(x_j) = 0 for all j∈{1,…,m}j \in \{1, \dots, m\}
    for i=1i = 1 to TT:
        for each j∈{1,…,m}j \in \{1, \dots, m\}:
            wj=1+(e(xj))4w_j = 1 + (e(x_j))^4
        Normalize weights such that ∑j=1mwj=m\sum_{j=1}^m w_j = m
        Construct S′S' by resampling mm instances from SS with probability proportional to ww
        Ci=I(S′)C_i = I(S')
        for each j∈{1,…,m}j \in \{1, \dots, m\}:
            if Ci(xj)≠yjC_i(x_j) \ne y_j:
                e(xj)=e(xj)+1e(x_j) = e(x_j) + 1
    for any test instance xx:
        C∗(x)=arg⁡max⁡y∈Y∑i:Ci(x)=y1C^*(x) = \arg\max_{y \in Y} \sum_{i: C_i(x) = y} 1
    return C∗C^*

    Arc-x4 requires resampling rather than passing instance weights directly to the base inducer (reweighting). On MC4 decision trees, Arc-x4 with resampling (Arc-x4-resample) matches AdaBoost with 9.81% average error (variance 4.0%), whereas Arc-x4 with reweighting (Arc-x4-reweight) degrades significantly to 10.86% average error due to high variance (4.9%). Final prediction is made by unweighted majority vote.

  5. Knowl 5 — Probabilistic Bagging with Unpruned Decision Trees

    model/method

    Probabilistic Bagging (p-Bagging) aggregates the continuous probability distributions output by individual ensemble models rather than voting discrete categorical predictions.

    Given TT bootstrap replicates S1′,…,ST′S'_1, \dots, S'_T sampled uniformly with replacement from training sample SS of size mm, an inducer builds classifiers C1,…,CTC_1, \dots, C_T. Each classifier CiC_i estimates a class probability distribution P(y∣x,Ci)P(y \mid x, C_i) for an instance xx and class label y∈Yy \in Y. The p-Bagging decision rule computes the unweighted average distribution across sub-classifiers and outputs the class with maximum average probability:

    C∗(x)=arg⁡max⁡y∈Y1T∑i=1TP(y∣x,Ci)C^*(x) = \arg\max_{y \in Y} \frac{1}{T} \sum_{i=1}^T P(y \mid x, C_i)

    When applied to top-down decision trees, p-Bagging performs substantially better when subtree pruning is disabled. Standard decision tree pruning replaces subtrees with leaves whenever all sibling nodes predict the same majority class, discarding fine-grained probability differences (e.g., pruning distinct children with 100%/0% and 60%/40% class ratios into a 70%/30% parent). Although individual unpruned trees have high variance, voting TT unpruned trees suppresses variance while eliminating the bias introduced by pruning. For MC4 trees, disabling pruning and adopting p-Bagging reduces average error from 10.4% to 10.2% and mean-squared error from 10.4% to 7.5%.

  6. Knowl 6 — Backfitting in Bagged Decision Trees

    model/method

    Backfitting is a post-processing procedure for bagged decision tree ensembles that recalibrates leaf probability distributions using the full training set without altering tree topologies.

    Because each standard bootstrap replicate contains only approximately 1−1/e≈63.2%1 - 1/e \approx 63.2\% of unique instances from the full training set SS, leaf probability estimates in unpruned trees are derived from limited data samples. After an unpruned decision tree CiC_i is constructed on bootstrap sample Si′S'_i, the entire original training set SS of mm instances is passed down CiC_i. The splits and structure of CiC_i remain fixed, but class frequency counts and probability estimates at every leaf are recomputed based on all instances in SS that fall into that leaf.

    Applying backfitting to unpruned probabilistic bagged MC4 decision trees (backfit-p-Bagging) reduces average classification error across 14 benchmark datasets from 10.4% to 10.1% (a 3% relative reduction). In bias-variance decomposition, backfitting maintains constant bias (6.7% vs. 6.6%) while reducing variance from 3.9% to 3.4% (an 11% relative variance reduction), with variance either decreasing or remaining unchanged on every individual dataset.

  7. Knowl 7 — Wagging (Weight Aggregation)

    model/method

    Wagging (Weight Aggregation) is a perturbation-based ensemble algorithm that perturbs continuous instance weights via additive Gaussian noise rather than drawing discrete bootstrap resamples.

    Given a training set S={(x1,y1),…,(xm,ym)}S = \{(x_1, y_1), \dots, (x_m, y_m)\}, each trial i∈{1,…,T}i \in \{1, \dots, T\} initializes uniform instance weights of 1 and perturbs each instance weight wjw_j by drawing from a zero-mean Gaussian distribution with standard deviation σ\sigma:

    wj=max⁡(0,1+N(0,σ2))w_j = \max\left(0, 1 + \mathcal{N}(0, \sigma^2)\right)

    The base inducer is trained on the continuously weighted sample SS. Wagging provides a direct mechanism to trade off bias and variance: increasing σ\sigma increases the proportion of instance weights that clamp to zero (effectively shrinking the active sample size), which increases inductive bias while decreasing model variance.

    On MC4 decision trees, Wagging with σ∈[2.0,3.0]\sigma \in [2.0, 3.0] achieves average classification errors of 10.19% (σ=2.0\sigma = 2.0), 10.16% (σ=2.5\sigma = 2.5), and 10.12% (σ=3.0\sigma = 3.0), matching the performance of unpruned probabilistic bagging (10.21%).

  8. Knowl 8 — Sensitivity of AdaBoost to Noise and Decision Tree Size Growth

    empirical result

    While Bagging never increases classification error relative to the base classifier across benchmark datasets, AdaBoost exhibits substantial performance degradation on noisy datasets, which strongly correlates with runaway growth in decision tree complexity:

    • Noise Sensitivity: On LED-24 (which contains 10% attribute noise), AdaBoost increases error by 3.1% absolute (from 34.1% for MC4 to 37.2% for AdaBoost MC4). When LED-24 attribute noise is systematically varied from 1% to 9%, AdaBoost consistently underperforms baseline MC4, with the performance deficit widening as noise increases (error increase of 0.84% at 1% noise, 0.88% at 2%, 1.01% at 3%, and 2.9% at 6% noise). AdaBoost also degrades error on noisy real-world datasets: Adult (15.0% to 16.3%), Hypothyroid (1.2% to 1.5%), and Sick-euthyroid (2.2% to 2.4%).

    • Correlation with Tree Growth: For datasets where AdaBoost significantly reduces error (waveform-40, satimage, shuttle), the average number of nodes per tree decreases relative to the standalone MC4 tree. Conversely, for all datasets where AdaBoost degrades performance, average tree size inflates dramatically: Hypothyroid trees grow from 10 to 25 nodes, Sick-euthyroid from 13 to 43 nodes, LED-24 from 114 to 179 nodes, and Adult from 776 to 2,513 nodes per tree. This indicates that AdaBoost overfits noise by repeatedly boosting the weights of mislabeled instances and outliers, forcing successive trees to grow elaborate subtrees to isolate them.

  9. Knowl 9 — Mean-Squared Error Reduction in Voting Classifiers

    empirical result

    Evaluating probability calibration using test-set Mean-Squared Error (MSE / Brier score), defined for an instance xx with true label ytruey_{\text{true}} as MSE=(1−P(C(x)=ytrue))2\text{MSE} = (1 - P(C(x) = y_{\text{true}}))^2, demonstrates that probabilistic ensemble voting produces superior probability estimates compared to single models:

    • For MC4 decision trees, single-tree deterministic predictions yield an average MSE of 10.4%. Standalone MC4 using leaf frequency probabilities yields 10.7% MSE, which drops to 10.0% MSE when using mm-estimate Laplace correction. Unpruned probabilistic bagging (p-Bagging) reduces average MSE to 7.5%—a 21% relative improvement over Laplace-corrected single trees and a 28% relative reduction compared to deterministic bagging.
    • For Naive-Bayes, average MSE drops from 13.1% (deterministic classification) to 9.8% under p-Bagging (a 24% relative reduction), outperforming standalone probability estimation (10.5% MSE).
    • For MC4(1)-disc, MSE drops from 31.1% to 18.4% under p-Bagging (a 34% relative reduction).

    In contrast, AdaBoost yields no meaningful MSE reduction over its deterministic zero-one classification error (difference <0.1%< 0.1\%) because boosting explicitly minimizes classification error and generates biased probability estimates due to iterative training set skewing.

  10. Knowl 10 — Numerical Instability and Underflow in Boosting Implementations

    limitation

    In sequential boosting algorithms such as AdaBoost.M1, instances that are classified correctly across multiple successive trials experience geometric weight decay. For a classifier error ϵi≈0\epsilon_i \approx 0, instance weights shrink by factors of approximately 2k2^k over kk correct classifications.

    In an ensemble of T=25T = 25 trials, weights of consistently correct instances decrease by factors up to 225≈3.35×1072^{25} \approx 3.35 \times 10^7, reaching values around 3×10−83 \times 10^{-8}. If an implementation enforces a standard numerical cutoff (such as removing instances with weight <10−6< 10^{-6}), nearly all correctly classified instances are pruned from the training set. On the Shuttle dataset, this underflow caused trial 6 to achieve 0% training error on a nearly empty dataset but an catastrophic test error of 60.86% (compared to 0.38% baseline).

    To prevent underflow without distorting the training distribution:

    1. Update weights using the direct formulation wj/(2ϵi)w_j / (2\epsilon_i) for misclassified and wj/(2(1−ϵi))w_j / (2(1-\epsilon_i)) for correctly classified instances to maintain sample sum conservation.
    2. Clamp weights that drop below numerical thresholds (e.g., 10−610^{-6}) to the threshold value rather than discarding instances.
    3. For large-scale boosting (T≥100T \ge 100), track log-weights and add a smoothing term 0.5/m0.5/m to the error formula numerator and denominator.

  11. Knowl 11 — Reduced-Error Pruning Inhibition from Bootstrap Instance Duplication

    empirical result

    In top-down decision tree induction (C4.5/MC4), unpruned trees grown on bootstrap replicates are smaller on average than unpruned trees grown on the full training dataset (average size of 496 nodes for Bagged replicate trees vs. 667 nodes for single full-data trees, a 25% reduction). This occurs because each bootstrap sample contains only ≈63.2%\approx 63.2\% unique instances, reflecting the known scaling of unpruned tree complexity with training set size.

    However, after reduced-error pruning is applied, the pruned bagged trees are larger than the pruned single trees (average size of 240 nodes vs. 198 nodes; on the Adult dataset, pruned bagged trees average 1,510 nodes compared to 776 nodes for single MC4).

    This structural divergence occurs because sampling with replacement produces duplicate instances within bootstrap replicates. Instance duplication artificially inflates leaf frequencies and makes observed training patterns appear statistically stronger to the reduced-error pruning criterion, which inhibits subtree collapse and causes the algorithm to retain nodes that would otherwise be pruned as noise on non-replicated data.

  12. Knowl 12 — Benchmark Dataset Characteristics and Controlled Training Sizes for Ensemble Evaluation

    data/table

    To evaluate voting algorithms under conditions where confidence intervals remain narrow and error reduction is not precluded by Bayes-optimal saturation, 14 datasets from the UC Irvine repository were selected with at least 1,000 instances. Training set sizes were determined by analyzing empirical learning curves, selecting points along the downward slope while reserving at least 50% of instances for test evaluation.

    Data set Dataset size Training set size Continuous attrs Nominal attrs Classes
    Credit (German) 1,000 300 7 13 2
    Image segmentation (segment) 2,310 500 19 0 7
    Hypothyroid 3,163 1,000 7 18 2
    Sick-euthyroid 3,163 800 7 18 2
    DNA 3,186 500 0 60 3
    Chess 3,196 500 0 36 2
    LED-24 3,200 500 0 24 10
    Waveform-40 5,000 1,000 40 0 3
    Satellite image (satimage) 6,435 1,500 36 0 7
    Mushroom 8,124 1,000 0 22 2
    Nursery 12,960 3,000 0 8 5
    Letter 20,000 5,000 16 0 26
    Adult 48,842 11,000 6 8 2
    Shuttle 58,000 5,000 9 0 7

    All ensemble comparisons used a standardized ensemble size of T=25T = 25 sub-classifiers, which balances asymptotic convergence with practical computational limits for multi-model induction.

Coverage note — None was omitted; all primary algorithms (AdaBoost.M1, Arc-x4, Bagging), variants (p-Bagging, backfitting, Wagging), bias-variance decompositions, and empirical findings on noise, tree growth, and probability estimation are fully covered.

References

  1. 1.Ali, K.M. (1996). Learning probabilistic relational concept descriptions. Ph.D. thesis, University of California, Irvine. http://www.ics.uci.edu/~ali.
  2. 2.Becker, B., Kohavi, R., & Sommerfield, D. (1997). Visualizing the simple bayesian classifier. KDD Workshop on Issues in the Integration of Data Mining and Data Visualization.
  3. 3.Bernardo, J.M., & Smith, A.F. (1993). Bayesian theory. John Wiley & Sons.
  4. 4.Breiman, L. (1994). Heuristics of instability in model selection (Technical Report). Berkeley: Statistics Department, University of California.
  5. 5.Breiman, L. (1996a). Arcing classifiers (Technical Report). Berkeley: Statistics Department, University of California. http://www.stat.Berkeley.EDU/users/breiman/.
  6. 6.Breiman, L. (1996b). Bagging predictors. Machine Learning, 24, 123–140.
  7. 7.Breiman, L. (1997). Arcing the edge (Technical Report 486). Berkeley: Statistics Department, University of California. http://www.stat.Berkeley.EDU/users/breiman/.
  8. 8.Buntine, W. (1992a). Learning classification trees. Statistics and Computing, 2(2), 63–73.
  9. 9.Buntine, W. (1992b). A theory of learning classification rules. Ph.D. thesis, University of Technology, Sydney, School of Computing Science.
  10. 10.Blake, C. Keogh, E., & Merz, C.J. (1998). UCI repository of machine learning databases. http://www.ics. uci.edu/~mlearn/MLRepository.html.
  11. 11.Cestnik, B. (1990). Estimating probabilities: A crucial task in machine learning. In L.C. Aiello (Ed.), Proceedings of the Ninth European Conference on Artificial Intelligence (pp. 147–149).
  12. 12.Chan, P., Stolfo, S., & Wolpert, D. (1996). Integrating multiple learned models for improving and scaling machine learning algorithms. AAAI Workshop.
  13. 13.Craven, M.W., & Shavlik, J.W. (1993). Learning symbolic rules using artificial neural networks. Proceedings of the Tenth International Conference on Machine Learning (pp. 73–80). Morgan Kaufmann.
  14. 14.Dietterich, T.G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7).
  15. 15.Dietterich, T.G., & Bakiri, G. (1991). Error-correcting output codes: A general method for improving multiclass inductive learning programs. Proceedings of the Ninth National Conference on Artificial Intelligence (AAAI-91) (pp. 572–577).
  16. 16.Domingos, P. (1997). Why does bagging work? A Bayesian account and its implications. In D. Heckerman, H. Mannila, D. Pregibon, & R. Uthurusamy (Eds.), Proceedings of the Third International Conference on Knowledge Discovery and Data Mining (pp. 155–158). AAAI Press.
  17. 17.Domingos, P., & Pazzani, M. (1997). Beyond independence: Conditions for the optimality of the simple Bayesian classifier. Machine Learning, 29(2/3), 103–130.
  18. 18.Drucker, H., & Cortes, C. (1996). Boosting decision trees. Advances in neural information processing systems 8’ (pp. 479–485).
  19. 19.Duda, R., & Hart, P. (1973). Pattern classification and scene analysis. Wiley.
  20. 20.Efron, B., & Tibshirani, R. (1993). An introduction to the bootstrap. Chapman & Hall.
  21. 21.Elkan, C. (1997). Boosting and naive bayesian learning (Technical Report). San Diego: Department of Computer Science and Engineering, University of California.
  22. 22.Fayyad, U.M., & Irani, K.B. (1993). Multi-interval discretization of continuous-valued attributes for classification learning. Proceedings of the 13th International Joint Conference on Artificial Intelligence (pp. 1022–1027). Morgan Kaufmann Publishers.
  23. 23.Freund, Y. (1990). Boosting a weak learning algorithm by majority. Proceedings of the Third Annual Workshop on Computational Learning Theory (pp. 202–216).
  24. 24.Freund, Y. (1996). Boosting a weak learning algorithm by majority. Information and Computation, 121(2), 256– 285.
  25. 25.Freund, Y., & Schapire, R.E. (1995). A decision-theoretic generalization of on-line learning and an application to boosting. Proceedings of the Second European Conference on Computational Learning Theory (pp. 23–37). Springer-Verlag, To appear in Journal of Computer and System Sciences.
  26. 26.Freund, Y., & Schapire, R.E. (1996). Experiments with a new boosting algorithm. In L. Saitta (Ed.), Machine Learning: Proceedings of the Thirteenth National Conference (pp. 148–156). Morgan Kaufmann.
  27. 27.Friedman, J.H. (1997). On bias, variance, 0/1-loss, and the curse of dimensionality. Data Mining and Knowledge Discovery, 1(1), 55–77. ftp://playfair.stanford.edu/pub/friedman/curse.ps.Z.
  28. 28.Geman, S., Bienenstock, E., & Doursat, R. (1992). Neural networks and the bias/variance dilemma. Neural Computation, 4, 1–48.
  29. 29.Good, I.J. (1965). The estimation of probabilities: An essay on modern bayesian methods. M.I.T. Press.
  30. 30.Holte, R.C. (1993). Very simple classification rules perform well on most commonly used datasets. Machine Learning, 11, 63–90.
  31. 31.Iba, W., & Langley, P. (1992). Induction of one-level decision trees. Proceedings of the Ninth International Conference on Machine Learning (pp. 233–240). Morgan Kaufmann Publishers.
  32. 32.Kohavi, R. (1995a). A study of cross-validation and bootstrap for accuracy estimation and model selection. In C.S. Mellish (Ed.), Proceedings of the 14th International Joint Conference on Artificial Intelligence (pp. 1137–1143). Morgan Kaufmann. http://robotics.stanford.edu/~ronnyk.
  33. 33.Kohavi, R. (1995b). Wrappers for performance enhancement and oblivious decision graphs. Ph.D. thesis, Stanford University, Computer Science department. STAN-CS-TR-95-1560. http://robotics.Stanford.EDU/~ ronnyk/teza.ps.Z.
  34. 34.Kohavi, R., Becker, B., & Sommerfield, D. (1997). Improving simple bayes. The Nineth European Conference on Machine Learning, Poster Papers’ (pp. 78–87). Available at http://robotics.stanford. edu/users/ronnyk.
  35. 35.Kohavi, R., & Kunz, C. (1997). Option decision trees with majority votes. In D. Fisher (Ed.), Machine Learning: Proceedings of the Fourteenth International Conference (pp. 161–169). Morgan Kaufmann Publishers. Available at http://robotics.stanford.edu/users/ronnyk.
  36. 36.Kohavi, R., & Sahami, M. (1996). Error-based and entropy-based discretization of continuous features. Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (pp. 114–119).
  37. 37.Kohavi, R., & Sommerfield, D. (1995). Feature subset selection using the wrapper model: Overfitting and dynamic search space topology. The First International Conference on Knowledge Discovery and Data Mining (pp. 192– 197).
  38. 38.Kohavi, R., Sommerfield, D., & Dougherty, J. (1997). Data mining using MLC++: A machine learning library in C++. International Journal on Artificial Intelligence Tools 6(4), 537–566. http://www.sgi.com/ Technology/mlc.
  39. 39.Kohavi, R., & Wolpert, D.H. (1996). Bias plus variance decomposition for zero-one loss functions. In L. Saitta (Ed.), Machine Learning: Proceedings of the Thirteenth International Conference (pp. 275–283). Morgan Kaufmann. Available at http://robotics.stanford.edu/users/ronnyk.
  40. 40.Kong, E.B., & Dietterich, T.G. (1995). Error-correcting output coding corrects bias and variance. In A. Prieditis & S. Russell (Eds.), Machine Learning: Proceedings of the Twelfth International Conference (pp. 313–321). Morgan Kaufmann.
  41. 41.Kwok, S.W., & Carter, C. (1990). Multiple decision trees. In R.D. Schachter, T.S. Levitt, L.N. Kanal, & J.F. Lemmer (Eds.), Uncertainty in Artificial Intelligence (pp. 327–335). Elsevier Science Publishers.
  42. 42.Langley, P., Iba, W., & Thompson, K. (1992). An analysis of Bayesian classifiers. Proceedings of the Tenth National Conference on Artificial Intelligence (pp. 223–228). AAAI Press and MIT Press.
  43. 43.Langley, P., & Sage, S. (1997). Scaling to domains with many irrelevant features. In R. Greiner (Ed.),Computational learning theory and natural learning systems (Vol. 4). MIT Press.
  44. 44.Oates, T., & Jensen, D. (1997). The effects of training set size on decision tree complexity. In D. Fisher (Ed.), Machine Learning: Proceedings of the Fourteenth International Conference (pp. 254–262). Morgan Kaufmann.
  45. 45.Oliver, J., & Hand, D. (1995). On pruning and averaging decision trees. In A. Prieditis & S. Russell (Eds.), Machine Learning: Proceedings of the Twelfth International Conference (pp. 430–437). Morgan Kaufmann.
  46. 46.Pazzani, M., Merz, C., Murphy, P., Ali, K., Hume, T., & Brunk, C. (1994). Reducing misclassification costs. Machine Learning: Proceedings of the Eleventh International Conference. Morgan Kaufmann.
  47. 47.Quinlan, J.R. (1993). C4.5: programs for machine learning. San Mateo, California: Morgan Kaufmann.
  48. 48.Quinlan, J.R. (1994). Comparing connectionist and symbolic learning methods. In S.J. Hanson, G.A. Drastal, & R.L. Rivest (Eds.), Computational learning theory and natural learning systems (Vol. I: Constraints and prospects, chap. 15, pp. 445–456). MIT Press.
  49. 49.Quinlan, J.R. (1996). Bagging, boosting, and c4.5. Proceedings of the Thirteenth National Conference on Artificial Intelligence (pp. 725–730). AAAI Press and the MIT Press.
  50. 50.Ridgeway, G., Madigan, D., & Richardson, T. (1998). Interpretable boosted naive bayes classification. Proceedings of the Fourth International Conference on Knowledge Discovery and Data Mining.
  51. 51.Schaffer, C. (1994). A conservation law for generalization performance. Machine Learning: Proceedings of the Eleventh International Conference (pp. 259–265). Morgan Kaufmann.
  52. 52.Schapire, R.E. (1990). The strength of weak learnability. Machine Learning, 5(2), 197–227.
  53. 53.Schapire, R.E., Freund, Y., Bartlett, P., & Lee, W.S. (1997). Boosting the margin: A new explanation for the effectiveness of voting methods. In D. Fisher (Ed.), Machine Learning: Proceedings of the Fourteenth International Conference (pp. 322–330). Morgan Kaufmann.
  54. 54.Wolpert, D.H. (1992). Stacked generalization. Neural Networks, 5, 241–259.
  55. 55.Wolpert, D.H. (1994). The relationship between PAC, the statistical physics framework, the Bayesian framework, and the VC framework. In D.H. Wolpert (Ed.), The mathematics of generalization. Addison Wesley.

Citation

MLA
Bauer, E., and R. Kohavi. “An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants”. Machine Learning, vol. 36, nos. 1-2, 1999, pp. 105–39, https://doi.org/10.1023/A:1007515423169.
APA
Bauer, E., & Kohavi, R. (1999). An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants. Machine Learning, 36(1-2), 105–139. https://doi.org/10.1023/A:1007515423169
Chicago
Bauer, E., and R. Kohavi. 1999. “An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants”. Machine Learning 36 (1-2): 105–39. https://doi.org/10.1023/A:1007515423169.
Harvard
Bauer, E. and Kohavi, R. (1999) “An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants”, Machine Learning, 36(1-2), pp. 105–139. Available at: https://doi.org/10.1023/A:1007515423169.
Vancouver
1. Bauer E, Kohavi R (1999) An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants. Machine Learning 36:105–139

BibTeX

@article{Bauer_1999, title={An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants}, volume={36}, ISSN={1573-0565}, url={http://dx.doi.org/10.1023/A:1007515423169}, DOI={10.1023/a:1007515423169}, number={1-2}, journal={Machine Learning}, publisher={Springer Science and Business Media LLC}, author={Bauer, Eric and Kohavi, Ron}, year={1999}, month=July, pages={105–139} }
Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF