Back

Neural Network Ensembles

L. K. HansenP. Salamon

article1990TPAMI2,658 citations

Demonstrates that combining independently trained neural networks through consensus voting substantially reduces generalization error by exploiting the distinct classification mistakes caused by different local minima.

Listen

Supervised neural networks are widely used for automated pattern recognition and classification, yet standard training practices often suffer from overtraining and fail to minimize real-world classification errors. Traditional workflows typically train multiple candidate models, select the single best performer, and discard the rest, leaving significant residual error. The article addresses how organizations can systematically measure real-world performance and reduce classification errors without continually increasing individual model complexity.

The article sets out to demonstrate that combining multiple neural networks into a voting ensemble significantly improves generalization accuracy compared to relying on any single optimal network. It also evaluates how cross-validation can be used to objectively optimize network architecture parameters, such as the number of hidden neurons.

To evaluate this approach, the authors developed analytical consensus models derived from fault-tolerant computing and validated them through controlled computational experiments. The empirical evaluation examined feed-forward networks applied to two synthetic classification benchmarks: a continuous non-linear decision problem and a 20-dimensional noisy pattern classification task. Across these setups, the authors tested different ensemble sizes, training configurations (identical versus independent training data), and architectural depths, comparing majority and plurality voting schemes against single-model baselines.

The analysis produced several key findings. First, voting ensembles consistently outperform single networks; in the continuous non-linear task, a consensus vote among three imperfect networks achieved a near-perfect decision boundary even though each individual network exhibited a 25% to 30% error rate. Second, ensembles of simpler, single-layer networks systematically outperformed larger, more complex two-layer networks with an equivalent total number of hidden units. Third, training individual networks on independent data subsets yielded markedly lower ensemble error rates than training all copies on a shared dataset. Fourth, the primary performance gains occur rapidly with modest group sizes, showing substantial error reduction when expanding from three to five models before benefits begin leveling off. Finally, mathematical modeling confirmed that incorporating weaker networks into an ensemble still reduces net error provided individual error rates remain below 50%.

These findings have direct operational and strategic implications. Organizations deploying machine learning can lower operational failure risk and improve system accuracy by retaining sub-optimal training runs rather than discarding them. Because an ensemble of simpler models outperforms complex individual architectures, teams can reduce software and architecture engineering costs while avoiding the severe over-fitting risks common to oversized networks. The results also show that diversity in training data and model initialization is a key asset that actively suppresses coincident system failures.

For practical implementation, technical teams should adopt cross-validation to select baseline network sizes and deploy ensembles of three to seven networks utilizing plurality or majority voting for production classification. Teams should actively foster model diversity by training ensemble members on distinct subsets of data and using varied initializations. Organizations do not need to discard slightly inferior candidate models generated during training, as integrating them into voting pools reliably improves net reliability.

Decision-makers should note that the analytical and experimental validations rely on synthetic model problems with controlled noise and known rules. While maximum-entropy models provided accurate performance predictions for independently trained networks, real-world data distributions may introduce higher levels of correlated error. Nevertheless, there is high confidence in the fundamental conclusion: combining independently trained neural network ensembles provides a robust, low-risk mechanism to suppress classification errors.

ANDREX Radiation Products A/SSan Diego State Universitybase classifiersclassifier ensemblesensemble classifiersNeural NetworkEnsembleCrossvalidation
Hansen et al (1990).pdf
  • Paper: Optimal Brain Damage, Yann LeCun et al. (1989). This paper establishes the foundational problem of overparameterization and network complexity control that neural network ensembles aim to overcome without individual model pruning.
Cover for Neural Network Ensembles

Abstract

We propose several means for improving the performance and training of neural networks for classification. We use crossvalidation as a tool for optimizing network parameters and architecture. We show further that the remaining residual “generalization” error can be reduced by invoking ensembles of similar networks.

Topics

base classifiersclassifier ensemblesensemble classifiersfeedforward neural networks

Keywords

Neural NetworkEnsembleCrossvalidationMajority VotingPlurality VotingLocal MinimumBackpropagationHidden NeuronMaximum EntropyFault Tolerant Computing

Table of Contents

  • I. INTRODUCTION
  • III. ENSEMBLES OF NEURAL NETWORKS
  • IV. THE PROBLEM OF MANY LOCAL MINIMA
  • A. How Local Minima Differ
  • B. Different Search Methods and Objectives
  • V. MODELS OF COLLECTIVE PERFORMANCE
  • A. Independent Errors
  • B. A Model Incorporating Input Difficulty
  • C. A Model Incorporating Network Proficiency
  • VI. EXPERIMENTS
  • A. Example 1: The Generalized XOR
  • B. Example 2: A Model Problem
  • VII. RESULTS
  • A. The Generalized XOR
  • B. Classification of Random Patterns
  • VIII. CONCLUSIONS

Knowls

  1. Knowl 1 — Majority Voting Error Rate under Independent Errors

    theoretical result

    For an ensemble of NN independently trained neural networks performing classification, assume each network independently makes an error with probability pp (and classifies correctly with probability 1−p1 - p). Under majority voting, the ensemble outputs the correct class if more than N/2N/2 networks choose the correct class; otherwise, an error occurs.

    The probability PmajP_{\text{maj}} that the majority decision makes an error is: Pmaj=∑k>N/2N(Nk)pk(1−p)N−kP_{\text{maj}} = \sum_{k > N/2}^{N} \binom{N}{k} p^k (1 - p)^{N-k} where kk is the number of networks that make an error.

    Key properties:

    • If p<1/2p < 1/2, PmajP_{\text{maj}} is strictly monotonically decreasing in NN for odd NN (or even NN). In the limit N→∞N \to \infty, Pmaj→0P_{\text{maj}} \to 0.
    • If p>1/2p > 1/2, increasing NN strictly increases the ensemble error rate, tending to 11 as N→∞N \to \infty.

  2. Knowl 2 — Plurality Voting Error Rate with Random Errors and Effective Confusion

    equation

    Consider an ensemble of NN independently trained networks classifying inputs into MM classes, where each network independently has an error probability pp. Under the "random errors" assumption, whenever a network errs, it chooses uniformly at random from among the remaining M−1M - 1 incorrect classes.

    Let n1∈{0,1,…,N}n_1 \in \{0, 1, \dots, N\} be the number of networks correctly identifying the true pattern. An error occurs under plurality voting if at least one incorrect class i∈{2,…,M}i \in \{2, \dots, M\} receives as many votes as the true class (ni≥n1n_i \ge n_1). The conditional probability that no incorrect class receives n1n_1 or more votes is:

    \sum_{k=0}^{M-1} (-1)^k \binom{M-1}{k} \frac{\binom{N - n_1(1 + k) + M - 2}{N - n_1(1 + k)}}{\binom{N - n_1 + M - 2}{M - 2}}, & \text{if } n_1 \ge 1 + \frac{N - 1}{M} \\ 0, & \text{if } n_1 < 1 + \frac{N - 1}{M} \end{cases}$$ The total ensemble failure probability under plurality voting, $P_{\text{plurality}}(p)$, is given by: $$P_{\text{plurality}}(p) = \sum_{n_1 = 0}^{\lfloor 1 + (N - 1)/M \rfloor} \binom{N}{n_1} (1 - p)^{n_1} p^{N - n_1} + \sum_{n_1 = \lceil 1 + (N - 1)/M \rceil}^{N} \binom{N}{n_1} (1 - p)^{n_1} p^{N - n_1} \left[ 1 - \sum_{k=0}^{M-1} (-1)^k \binom{M-1}{k} \frac{\binom{N - n_1(1 + k) + M - 2}{N - n_1(1 + k)}}{\binom{N - n_1 + M - 2}{M - 2}} \right]$$

  3. Knowl 3 — Ensemble Error Rate under Continuous Input Difficulty Distribution

    theoretical result

    When individual network errors are correlated due to varying pattern difficulty across input space, let θ(α)∈[0,1]\theta(\alpha) \in [0, 1] denote the failure probability of an infinite ensemble of networks trained on input pattern α\alpha. Let μ(θ)\mu(\theta) denote the input difficulty distribution, such that μ(θ) dθ\mu(\theta)\,d\theta is the fraction of inputs for which a fraction θ\theta of trained networks fail.

    The expected error rate of an ensemble of NN networks under majority voting is: Pmaj=∫01∑k>N/2N(Nk)θk(1−θ)N−kμ(θ) dθP_{\text{maj}} = \int_0^1 \sum_{k > N/2}^{N} \binom{N}{k} \theta^k (1 - \theta)^{N-k} \mu(\theta)\, d\theta

    For plurality voting, the expected ensemble error rate is: Pplurality=∫01Pplurality(θ) μ(θ) dθP_{\text{plurality}} = \int_0^1 P_{\text{plurality}}(\theta)\, \mu(\theta)\, d\theta

    In the asymptotic limit of infinitely many networks (N→∞N \to \infty), the majority voting error rate converges to the integral over inputs where the majority fails: lim⁡N→∞Pmaj=∫0.51μ(θ) dθ\lim_{N \to \infty} P_{\text{maj}} = \int_{0.5}^1 \mu(\theta)\, d\theta Majority consensus reduces the error rate relative to single networks whenever the bulk of the difficulty distribution μ(θ)\mu(\theta) lies in the interval θ<0.5\theta < 0.5.

  4. Knowl 4 — Maximum Entropy Prior for Input Difficulty Distribution

    model/method

    When the exact input difficulty distribution μ(θ)\mu(\theta) is not directly known but the average single-network error rate across inputs pˉ=∫01θμ(θ) dθ\bar{p} = \int_0^1 \theta \mu(\theta)\, d\theta is measured, the least-biased distribution μ∗(θ)\mu^*(\theta) on a discrete grid of difficulties θi=i/K\theta_i = i/K for i∈{0,1,…,K}i \in \{0, 1, \dots, K\} is obtained by maximizing entropy subject to the mean error constraint.

    The resulting distribution is of Boltzmann form: μ∗(θi)=e−λθi∑j=0Ke−λθj\mu^*(\theta_i) = \frac{e^{-\lambda \theta_i}}{\sum_{j=0}^{K} e^{-\lambda \theta_j}} where the Lagrange multiplier λ\lambda is determined uniquely by: ∑i=0Kθiμ∗(θi)=pˉ\sum_{i=0}^{K} \theta_i \mu^*(\theta_i) = \bar{p}

    This Boltzmann distribution μ∗\mu^* accurately predicts ensemble error rates when ensemble members are trained on independent training sets.

  5. Knowl 5 — Network Proficiency Screening Condition for Three-Network Ensembles

    theoretical result

    For an ensemble of three networks with individual error probabilities p1,p2,p3p_1, p_2, p_3 assuming independent errors, ordered such that p1≤p2,p3p_1 \le p_2, p_3 (making network 1 the best single network), majority consensus strictly outperforms the single best network (Pconsensus<p1P_{\text{consensus}} < p_1) if and only if: p1>p1p2p3+p1p2(1−p3)+p1p3(1−p2)+p2p3(1−p1)p_1 > p_1 p_2 p_3 + p_1 p_2 (1 - p_3) + p_1 p_3 (1 - p_2) + p_2 p_3 (1 - p_1)

    In the special case where the two auxiliary networks have identical error rates p2=p3=pp_2 = p_3 = p, the condition for majority consensus to improve upon p1p_1 is: p<p1−p1+32p13/2−2p12+O(p15/2)p < \sqrt{p_1} - p_1 + \frac{3}{2} p_1^{3/2} - 2 p_1^2 + \mathcal{O}(p_1^{5/2})

    Under this criterion, auxiliary networks with substantially higher individual error rates can still improve collective accuracy; for instance, if p1=0.10p_1 = 0.10, adding two networks with p2=p3=0.20p_2 = p_3 = 0.20 reduces the consensus error rate to 0.0980.098.

  6. Knowl 6 — Mechanisms Generating Diversity Across Neural Network Local Minima

    theoretical result

    Ensemble performance gains rely on individual networks failing on different subsets of the input space. In feed-forward networks trained via backpropagation, diversity among optimal parameter vectors ww arises from the complex local minima structure of the squared-error objective function:

    1. Symmetries in parameter space:

      • Discrete permutation symmetries: Permuting the hidden neuron indices preserves network output. Mixtures of symmetry-related configurations create frustrated states and a combinatorial explosion of distinct local minima.
      • Sign reflection symmetry: Inverting weights of antisymmetric activation functions yields distinct symmetry-related solutions.
      • Continuous saturation symmetry: Due to sigmoid saturation at large activation magnitudes, weights approaching a hidden unit can be scaled arbitrarily, forming continuous families of minima ("cones").

    2. Stochastic search dynamics:

      • Random initial weight assignments w(0)w^{(0)} and initial update velocities.
      • Random pattern presentation sequences during online gradient descent.
      • Gradient noise from single-pattern or mini-batch estimates, which acts as a stochastic thermal bath enabling uphill moves similar to simulated annealing.

  7. Knowl 7 — Ensemble Resolution of Generalization Failure on Continuous XOR

    empirical result

    On the generalized continuous XOR problem defined on the unit square [−0.5,0.5]×[−0.5,0.5][-0.5, 0.5] \times [-0.5, 0.5] with class label o(x,y)=sign(x⋅y)o(x, y) = \text{sign}(x \cdot y), single-hidden-layer feed-forward networks with 6 hidden neurons were trained via backpropagation with learning rates ηweights=0.05\eta_{\text{weights}} = 0.05, ηthresholds=0.01\eta_{\text{thresholds}} = 0.01, and momentum 0.990.99.

    Empirical findings:

    • Individual networks frequently get trapped in suboptimal local minima. Approximately 50% of trained single networks fail to learn the four-quadrant partition, exhibiting test error rates of 25% to 30% even after 30,000 presentations of training points.
    • A majority consensus decision over an ensemble of 3 independently trained networks consistently resolves the correct four-quadrant boundary, achieving near-zero generalization error on a test grid of 10,000 inputs.

  8. Knowl 8 — Effect of Independent vs Shared Training Sets on Ensemble Error Reduction

    empirical result

    On a 10-class problem on a 20-dimensional hypercube with bit-inversion noise ρ∈[0.0,0.20]\rho \in [0.0, 0.20], ensembles of 7 single-hidden-layer networks (10 hidden neurons each) were evaluated under two training data conditions:

    • Shared training set: All 7 networks were trained on the identical 100-sample training set with different initial weights and presentation orderings.
    • Independent training sets: Each network was trained on an independently generated 100-sample training set.

    While single-network average test error was identical across both regimes (e.g., ≈22%\approx 22\% at ρ=0.20\rho = 0.20), the ensemble trained on independent training sets achieved lower test error rates (plurality error ≈14%\approx 14\%) than the ensemble trained on shared data (plurality error ≈17%\approx 17\%). The empirical input difficulty distribution μ(θ)\mu(\theta) for independently trained networks matches the maximum entropy theoretical distribution, whereas shared data induces inter-network error correlations that limit ensemble gains.

  9. Knowl 9 — Performance Advantage of Ensembles over Deeper Multi-Layer Networks

    empirical result

    On a 10-class 20-dimensional hypercube classification task across bit-inversion noise levels ρ∈[0.0,0.20]\rho \in [0.0, 0.20], ensembles of single-hidden-layer networks were compared to deeper two-hidden-layer networks with matched or greater parameter counts:

    • An ensemble of 3 single-hidden-layer networks (10 hidden neurons each, 30 total hidden units) and an ensemble of 5 single-hidden-layer networks (50 total hidden units) consistently achieved lower classification error than single two-hidden-layer networks configured with 15×1515 \times 15 (30 hidden units) and 25×2525 \times 25 (50 hidden units).
    • At ρ=0.20\rho = 0.20 noise, the single 25×2525 \times 25 two-layer network achieved an average error of ≈20%\approx 20\%, whereas an ensemble of 5 single-layer networks achieved ≈14%\approx 14\% error.
    • Forming an 11-member ensemble of two-hidden-layer (25×2525 \times 25) networks further reduced classification error to ≈11%\approx 11\%.

  10. Knowl 10 — Estimation of Effective Degree of Confusion in Plurality Ensembles

    model/method

    In classification with MM classes, neural network errors are typically concentrated among a subset of confusion classes rather than spread uniformly across all M−1M - 1 alternatives. To model this, an effective degree of confusion parameter Meff≤MM_{\text{eff}} \le M is used in the plurality voting formula.

    MeffM_{\text{eff}} is estimated empirically by matching the observed ratio of plurality-to-majority performance improvement with the theoretical prediction. In the 10-class random pattern experiment (M=10M = 10), the estimated values are:

    • Noise level ρ=5%\rho = 5\%: Meff≈4M_{\text{eff}} \approx 4
    • Noise level ρ=10%\rho = 10\%: Meff≈5M_{\text{eff}} \approx 5
    • Noise level ρ=15%\rho = 15\%: Meff≈7M_{\text{eff}} \approx 7
    • Noise level ρ=20%\rho = 20\%: Meff≈9M_{\text{eff}} \approx 9

    As input noise increases, patterns become less distinguishable, causing errors to distribute more uniformly across all classes and driving MeffM_{\text{eff}} toward MM.

Coverage note — All substantial theoretical, architectural, and experimental contributions from the paper have been represented as knowls.

References

  1. 1.E. Levin, N. Tishby, and S. Solla, "A statistical approach to learning and generalization in layered neural networks," Proc. IEEE (Special Issue on Neural Networks), C. Lau, Guest Ed., 1990, to be published.
  2. 2.D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning internal representations by error propagation," in Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Vol. 1: Foundations, D. E. Rumelhart and J. L. McClelland, Eds. Cambridge, MA: MIT Press, 1986.
  3. 3.D. H. Ackley, G. E. Hinton, and T. J. Sejnowski, "A learning algorithm for Boltzmann machines," Cognitive Sci., vol. 9, pp. 147-169, 1985.
  4. 4.B. Widrow and R. Winter, "Neural nets for adaptive filtering and adaptive pattern recognition," Computer, vol. 25, Mar. 1988.
  5. 5.F. J. Pineda, "Generalization of backpropagation to recurrent neural networks," Phys. Rev. Lett., vol. 59, pp. 2229-2232, 1987.
  6. 6.T. J. Sejnowski and C. R. Rosenberg, "Parallel networks that learn to pronounce English text," Complex Syst., vol. 1, pp. 145-168, 1987.
  7. 7.D. E. Eckhardt, Jr., and L. D. Lee, "A theoretical basis for the analysis of multiversion software subject to coincident errors," IEEE Trans. Software Eng., vol. SE-11, pp. 1511-1517, 1985.
  8. 8.G. T. Toussaint, "Bibliography on estimation of misclassification," IEEE Trans. Inform. Theory, vol. IT-20, pp. 472-479, 1974.
  9. 9.N. Qian and T. J. Sejnowski, "Predicting the secondary structure of globular proteins using neural network models," J. Molecular Biol., vol. 202, pp. 865-884, 1989.
  10. 10.H. Bohr, J. Bohr, S. Brunak, R. M. J. Cotterill, B. Lautrup, L. Nørskov, O. H. Olsen, and S. B. Petersen, "Protein secondary structure and homology by neural networks," Fed. European Biochem. Soc. Lett., vol. 241, pp. 223-228, 1988.
  11. 11.R. P. Lippmann, "An introduction to computing with neural nets," IEEE ASSP Mag., pp. 4-22, Apr. 1987.
  12. 12.R. G. Palmer, "Broken ergodicity," Advances Phys., vol. 31, pp. 669-735, 1982.
  13. 13.S. Kirkpatrick, C. D. Gelatt, Jr., and M. P. Vecchi, "Optimization by simulated annealing," Science, vol. 220, pp. 671-680, 1983.
  14. 14.T. Schneider and E. Stoll, "Molecular-dynamics study of a three dimensional one-component model for distortive phase transitions," Phys. Rev. B, vol. 17, p. 1302, 1978.
  15. 15.J. Riordan, An Introduction to Combinatorial Analysis. Princeton, NJ: Princeton University Press, 1980.
  16. 16.F. Rosenblatt, Principles of Neurodynamics. New York: Spartan, 1959.

Citation

MLA
Hansen, L. K., and P. Salamon. “Neural Network Ensembles”. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, no. 10, 1990, pp. 993–1001, https://doi.org/10.1109/34.58871.
APA
Hansen, L. K., & Salamon, P. (1990). Neural network ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(10), 993–1001. https://doi.org/10.1109/34.58871
Chicago
Hansen, L. K., and P. Salamon. 1990. “Neural Network Ensembles”. IEEE Transactions on Pattern Analysis and Machine Intelligence 12 (10): 993–1001. https://doi.org/10.1109/34.58871.
Harvard
Hansen, L.K. and Salamon, P. (1990) “Neural network ensembles”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(10), pp. 993–1001. Available at: https://doi.org/10.1109/34.58871.
Vancouver
1. Hansen LK, Salamon P (1990) Neural network ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence 12:993–1001

BibTeX

@article{Hansen_1990, title={Neural network ensembles}, volume={12}, ISSN={0162-8828}, url={http://dx.doi.org/10.1109/34.58871}, DOI={10.1109/34.58871}, number={10}, journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, publisher={Institute of Electrical and Electronics Engineers (IEEE)}, author={Hansen, L.K. and Salamon, P.}, year={1990}, pages={993–1001} }
Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF