Neural Network Ensembles, Cross Validation, and Active Learning

Anders KroghJesper Vedelsby

article1994NeurIPS2,366 citations

Establishes the fundamental ambiguity decomposition relating ensemble generalization error directly to individual network errors and member disagreement, providing a principled framework to optimize ensemble weights and guide active learning using unlabeled data.

Listen

Acquiring labeled training data for machine learning models is frequently expensive, complex, and time-consuming. While combining multiple models into an ensemble generally improves predictive accuracy, organizations often struggle to evaluate overall error, assign proper importance to individual models, and prioritize which data points to label next.

The article demonstrates how measuring model disagreement across unlabeled data can be used to reliably estimate ensemble error, optimize individual model combination weights, and direct data acquisition through active learning.

The authors established a mathematical relationship showing that overall ensemble prediction error equals the average error of individual models minus their ambiguity, which represents the variance or disagreement among their predictions. They validated this framework using experiments on five-member neural network ensembles tasked with approximating a continuous function. The analysis evaluated ensemble performance across varied cross-validation partitioning sizes, tested quadratic optimization techniques to calculate optimal member weights using unlabeled data, and benchmarked an active learning scheme where points of highest model disagreement were sequentially selected from a candidate pool of unlabeled samples.

The investigation produced three key findings. First, mathematical and empirical results confirm that ensemble error is always lower than or equal to the average error of individual models, meaning higher disagreement among accurate models directly reduces overall error. Second, optimizing member weights via unlabeled data and cross-validation estimates significantly reduced prediction error compared to simple uniform averaging, dropping error closer to theoretical lower bounds. Third, the active learning strategy achieved drastically lower prediction errors with far fewer training samples than passive random sampling, while also demonstrating substantially lower variance across experimental trials.

These findings indicate that organizations can leverage inexpensive, abundant unlabeled data to enhance model performance, avoid overfitting, and optimize labeling budgets. By querying only the data points where models actively disagree, teams can cut the time and financial costs required to build accurate predictive systems.

Decision-makers implementing ensemble systems should adopt weighted model combinations using unlabeled data and employ ambiguity-based active learning to guide data collection. Before widespread deployment, organizations should conduct pilot testing, as the performance gains from specific cross-validation splits can vary depending on system setup and data distributions. The theoretical findings apply broadly to any continuous predictive models, though practical validation on more complex, multidimensional datasets is advised.

Nordita Technical University of Denmark active learning cross validation ensemble classifiers Active Learning Ensemble Generalization Error

Krogh et al (1994).pdf

Paper: Neural Network Ensembles, L. K. Hansen et al. (1990). Read this foundational paper first to understand how combining neural networks into voting ensembles improves generalization accuracy.

Paper: Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles, Balaji Lakshminarayanan et al. (2017). Read this book next to see how modern deep learning scales ensemble methods for reliable predictive uncertainty estimation.
Paper: Popular Ensemble Methods: An Empirical Study, David Opitz et al. (1999). Read this study next to explore empirical evaluations of bagging and boosting across diverse datasets and base classifiers.

Cover for Neural Network Ensembles, Cross Validation, and Active Learning

Abstract

Learning of continuous valued functions using neural network ensembles (committees) can give improved accuracy, reliable estimation of the generalization error, and active learning. The ambiguity is defined as the variation of the output of ensemble members averaged over unlabeled data, so it quantifies the disagreement among the networks. It is discussed how to use the ambiguity in combination with cross-validation to give a reliable estimate of the ensemble generalization error, and how this type of ensemble cross-validation can sometimes improve performance. It is shown how to estimate the optimal weights of the ensemble members using unlabeled data. By a generalization of query by committee, it is finally shown how the ambiguity can be used to select new training data to be labeled in an active learning scheme.

Topics

active learning cross validation ensemble classifiers neural networks validation value functions

Keywords

Active Learning Ensemble Generalization Error Neural Network Networks Neural Network Ensembles Network Ensembles Cross Ensembles Cross Validation Ambiguity

1 INTRODUCTION
2 THE BIAS-VARIANCE TRADEOFF
3 THE CROSS-VALIDATION ENSEMBLE
4 OPTIMAL WEIGHTS
5 ACTIVE LEARNING
6 CONCLUSION
Acknowledgements
References

Knowls

Knowl 1 — Ensemble Ambiguity Decomposition of Generalization Error
theoretical result
For an ensemble of $N$ neural networks (or other regression models) approximating a target function $f: \mathbb{R}^d \to \mathbb{R}$ under an input probability distribution $p(x)$ , let $V^\alpha(x)$ denote the output of member $\alpha \in \{1, \dots, N\}$ on input $x \in \mathbb{R}^d$ . The weighted ensemble prediction is:

$\overline{V}(x) = \sum_{\alpha=1}^N w_\alpha V^\alpha(x)$

where the weights are constrained such that $w_\alpha \ge 0$ and $\sum_{\alpha=1}^N w_\alpha = 1$ .

The generalization error and ambiguity for individual network $\alpha$ are defined respectively as:

$E^\alpha = \int dx \, p(x) \left( f(x) - V^\alpha(x) \right)^2$

$A^\alpha = \int dx \, p(x) \left( V^\alpha(x) - \overline{V}(x) \right)^2$

and the ensemble generalization error is $E = \int dx \, p(x) \left( f(x) - \overline{V}(x) \right)^2$ .

The ensemble generalization error decomposes exactly as:

$E = \overline{E} - \overline{A}$

where $\overline{E} = \sum_{\alpha=1}^N w_\alpha E^\alpha$ is the weighted average of the individual generalization errors, and $\overline{A} = \sum_{\alpha=1}^N w_\alpha A^\alpha$ is the ensemble ambiguity.

A fundamental property of this decomposition is that the ensemble ambiguity $\overline{A}$ depends only on the predictions of the ensemble members and can be calculated entirely from unlabeled data sampled from $p(x)$ , without requiring knowledge of the true target function $f(x)$ .
Knowl 2 — Pointwise Ambiguity Identity and Ensemble Error Bounds
theoretical result
For an ensemble of $N$ estimators $V^\alpha(x)$ combined via convex weights $w_\alpha \ge 0$ with $\sum_{\alpha=1}^N w_\alpha = 1$ , the ensemble prediction is $\overline{V}(x) = \sum_{\alpha=1}^N w_\alpha V^\alpha(x)$ . At any individual input point $x$ , the quadratic ensemble error $e(x) = (f(x) - \overline{V}(x))^2$ satisfies the exact pointwise identity:

$e(x) = \overline{\epsilon}(x) - \overline{a}(x)$

where $\overline{\epsilon}(x) = \sum_{\alpha=1}^N w_\alpha (f(x) - V^\alpha(x))^2$ is the weighted average error of the individual networks, and $\overline{a}(x) = \sum_{\alpha=1}^N w_\alpha (V^\alpha(x) - \overline{V}(x))^2$ is the ensemble ambiguity at $x$ .

Because the ensemble ambiguity is non-negative ( $\overline{a}(x) \ge 0$ ), this decomposition implies two bounds:
1. The ensemble quadratic error never exceeds the weighted average of individual errors:
$e(x) \le \overline{\epsilon}(x) \implies E \le \overline{E}$

For uniform weights $w_\alpha = 1/N$ , this yields $E \le \frac{1}{N}\sum_{\alpha=1}^N E^\alpha$ .
1. The ensemble ambiguity is a pointwise lower bound on the weighted average individual error:
$\overline{\epsilon}(x) \ge \overline{a}(x)$
Knowl 3 — Quadratic Optimization of Ensemble Weights via Unlabeled Data
model/method
The ensemble generalization error $E$ can be minimized over the weight vector $\mathbf{w} = (w_1, \dots, w_N)^\top$ subject to the constraints $w_\alpha \ge 0$ for all $\alpha$ and $\sum_{\alpha=1}^N w_\alpha = 1$ .

Using the decomposition $E = \overline{E} - \overline{A}$ , the objective function is expressed in terms of individual generalization errors $E^\alpha$ and the symmetric correlation matrix $C \in \mathbb{R}^{N \times N}$ defined by:

$C_{\alpha\beta} = \int dx \, p(x) V^\alpha(x) V^\beta(x)$

Using $\sum_{\alpha=1}^N w_\alpha = 1$ , the ensemble error becomes:

$E = \sum_{\alpha=1}^N w_\alpha E^\alpha + \sum_{\alpha=1}^N \sum_{\beta=1}^N w_\alpha C_{\alpha\beta} w_\beta - \sum_{\alpha=1}^N w_\alpha C_{\alpha\alpha}$

Because $C_{\alpha\beta}$ depends solely on network outputs and the input distribution $p(x)$ , it can be estimated to arbitrary precision from unlabeled data. Given estimates of individual errors $E^\alpha$ (e.g., from cross-validation holdout sets), finding the optimal ensemble weights is solved as a constrained quadratic optimization problem via quadratic programming or linear programming.
Knowl 4 — Optimality Condition for Non-Zero Ensemble Weights
theoretical result
Let $E = \overline{E} - \overline{A}$ be the ensemble generalization error subject to the unit simplex constraints $w_\alpha \ge 0$ and $\sum_{\alpha=1}^N w_\alpha = 1$ . At any constrained minimum of $E$ with respect to $\mathbf{w}$ , setting the derivative to zero under the constraints yields the optimality condition for each member $\alpha \in \{1, \dots, N\}$ :

$E^\alpha - A^\alpha = E \quad \text{or} \quad w_\alpha = 0$

where $E^\alpha = \int dx \, p(x)(f(x) - V^\alpha(x))^2$ is the individual generalization error of member $\alpha$ , $A^\alpha = \int dx \, p(x)(V^\alpha(x) - \overline{V}(x))^2$ is its individual ambiguity, and $E$ is the total ensemble generalization error.

This condition states that every network assigned a non-zero weight ( $w_\alpha > 0$ ) must contribute an identical difference $E^\alpha - A^\alpha$ equal to the overall ensemble error $E$ . A network whose generalization is poor or whose predictions are strongly redundant with the rest of the ensemble is assigned optimal weight $w_\alpha = 0$ .
Knowl 5 — Ambiguity-Based Active Learning for Regression
algorithm
For continuous-valued regression, ensemble ambiguity provides a query criterion that generalizes query-by-committee from classification to regression. Since ensemble ambiguity $\overline{a}(x) = \sum_{\alpha=1}^N w_\alpha (V^\alpha(x) - \overline{V}(x))^2$ is a pointwise lower bound on the weighted average individual squared error $\overline{\epsilon}(x)$ , candidate inputs with large ambiguity indicate regions in input space where network uncertainty and potential error are highest.
Input: Initial labeled training set $D = \{(x^\mu, y^\mu)\}_{\mu=1}^{p_0}$ , candidate pool size $M$ , target function $f$ , total query iterations $T$ , ensemble size $N$ , weights $\{w_\alpha\}_{\alpha=1}^N$ summing to 1
Output: Trained ensemble prediction $\overline{V}(x) = \sum_{\alpha=1}^N w_\alpha V^\alpha(x)
for iteration $t = 1$ to $T$ do
    Train/update ensemble members $V^1(x), \dots, V^N(x)$ independently on $D$
    Sample a candidate set of $M$ unlabeled inputs $\{x_j\}_{j=1}^M$ from input distribution $p(x)$
    for each candidate $x_j$ do
        Compute ensemble mean: $\overline{V}(x_j) = \sum_{\alpha=1}^N w_\alpha V^\alpha(x_j)$
        Compute ensemble ambiguity: $\overline{a}(x_j) = \sum_{\alpha=1}^N w_\alpha (V^\alpha(x_j) - \overline{V}(x_j))^2$
    end for
    Select the query point with maximum disagreement: $x^* = \arg\max_{x_j} \overline{a}(x_j)$
    Query label: $y^* = f(x^*)$
    Augment dataset: $D \leftarrow D \cup \{(x^*, y^*)\}$
end for
Train final ensemble members $V^1(x), \dots, V^N(x)$ on $D$
return $\overline{V}(x) = \sum_{\alpha=1}^N w_\alpha V^\alpha(x)$
In experiments on 1D continuous functions, $M = 800$ random candidates were evaluated at each step, and queries were added one at a time starting from $p_0 = 1$ initial training point.
Knowl 6 — Cross-Validation Ensembling for Diversity and Error Estimation
model/method
To simultaneously induce disagreement (increasing ensemble ambiguity $\overline{A}$ ) and obtain reliable estimates of individual generalization errors $E^\alpha$ without needing an external test set, a cross-validation ensembling scheme is structured as follows:
1. From a dataset of $p$ available examples, choose a holdout size $K \le p$ .
2. For each network $\alpha \in \{1, \dots, N\}$ in the ensemble, hold out a validation set of $K$ examples and train network $\alpha$ exclusively on the remaining $p - K$ examples.
3. Construct the $N$ holdout sets to have minimal mutual overlap. In particular, choosing $K \le p/N$ allows the $N$ holdout sets to be completely disjoint partitions of the training data.
This method serves two simultaneous roles:
- Forcing each network to train on a different data subset increases ensemble ambiguity $\overline{A}$ .
- The held-out $K$ examples for network $\alpha$ provide an unbiased cross-validation estimate $E^\alpha_{\text{CV}}$ , enabling the ensemble generalization error to be estimated as $E \approx \sum_{\alpha=1}^N w_\alpha E^\alpha_{\text{CV}} - \overline{A}$ , where $\overline{A}$ is estimated using unlabeled data.
Knowl 7 — Empirical Performance of Ambiguity-Based Active Learning
empirical result
Ambiguity-based active learning was evaluated against passive (random) sampling on learning a 1D square wave function $f(x)$ over the interval $[-2, 2]$ .
- Setup: Ensembles of $N = 5$ feed-forward neural networks with 20 hidden units each were trained independently using backpropagation. Both active and passive tests began with a single training example ( $p_0 = 1$ ) and added examples incrementally up to 50 examples. Active learning selected the candidate with maximum ensemble ambiguity $\overline{a}(x)$ out of $M = 800$ random unlabeled inputs per step. Evaluations were averaged over $2 \times 40$ independent runs for each training set size.
- Results: Active learning produced a significantly lower average generalization error than passive learning across all training set sizes from 5 to 50. In the range of 15 to 25 training examples, active learning reduced generalization error by over 50% relative to passive learning. Furthermore, active learning exhibited substantially lower variance (scatter) across independent runs, indicating that ambiguity-guided queries consistently produce more stable training sets for neural network ensembles.
Knowl 8 — Empirical Verification of Cross-Validation Error Estimation and Weight Optimization
empirical result
An ensemble of $N = 5$ feed-forward neural networks (20 hidden units each) was trained on $p = 200$ examples to approximate a 1D square wave function across holdout sizes $K \in [0, 80]$ (trained on $200 - K$ examples per network), with performance evaluated over 1000 independent test inputs and averaged over 12 runs.
- Error Estimation: The cross-validation estimate of ensemble error, computed via $E \approx \frac{1}{N}\sum_{\alpha=1}^N E^\alpha_{\text{CV}} - \overline{A}$ using held-out sets of size $K$ and unlabeled ambiguity $\overline{A}$ , closely matched the true generalization error across all evaluated values of $K$ .
- Weight Optimization: Finding optimal weights $w_\alpha$ via quadratic programming using cross-validation estimates $E^\alpha_{\text{CV}}$ and the unlabeled correlation matrix $C_{\alpha\beta}$ systematically improved performance over uniform weighting ( $w_\alpha = 1/5$ ). At $K = 40$ , optimal weighting decreased generalization error from approximately $0.042$ (uniform weights) to approximately $0.033$ , closely tracking the theoretical best possible error (approximately $0.025$ ) obtained when true individual errors $E^\alpha$ are known exactly.
Knowl 9 — Limitations of Ambiguity as an Error Proxy and Cross-Validation Generalization Tradeoff
limitation
Two key limitations affect the practical application of the ambiguity framework:
1. Asymmetric Error Indication: While high ensemble ambiguity $\overline{a}(x)$ guarantees a large weighted average individual error (since $\overline{\epsilon}(x) \ge \overline{a}(x)$ ), low ambiguity does not guarantee low generalization error. A strongly biased ensemble whose members make identical incorrect predictions outside the training set will exhibit near-zero ambiguity despite having high generalization error.
2. Ambiguity vs. Training Set Size Tradeoff: In cross-validation ensembling, holding out $K$ examples increases individual errors $E^\alpha$ (due to training on fewer examples, $p - K$ ) while increasing ambiguity $\overline{A}$ . Whether this net tradeoff reduces total ensemble error $E = \overline{E} - \overline{A}$ relative to $K=0$ depends on network convergence and dataset properties; a strict drop in generalization error over $K=0$ is not universally guaranteed.

Coverage note — No substantial contributed material was omitted; all primary theoretical results (ambiguity decomposition and weight optimality conditions), algorithmic procedures (ambiguity-based active learning), methodological frameworks (cross-validation ensembling and quadratic weight optimization), empirical benchmarks, and stated limitations are fully represented.

References

1.L.K. Hansen and P Salamon. Neural network ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(10):993-1001, Oct. 1990.
2.D.H Wolpert. Stacked generalization. Neural Networks, 5(2):241-59, 1992.
3.Michael P. Perrone and Leon N Cooper. When networks disagree: Ensemble method for neural networks. In R. J. Mammone, editor, Neural Networks for Speech and Image processing. Chapman-Hall, 1993.
4.S. Geman, E. Bienenstock, and R Doursat. Neural networks and the bias/variance dilemma. Neural Computation, 4(1):1-58, Jan. 1992.
5.Ronny Meir. Bias, variance and the combination of estimators; the case of linear least squares. Preprint (In Neuroprose), Technion, Heifa, Israel, 1994.
6.H.S. Seung, M. Opper, and H. Sompolinsky. Query by committee. In Proceedings of the Fifth Workshop on Computational Learning Theory, pages 287-294, San Mateo, CA, 1992. Morgan Kaufmann.
7.Y. Freund, H.S. Seung, E. Shamir, and N. Tishby. Information, prediction, and query by committee. In Advances in Neural Information Processing Systems, volume 5, San Mateo, California, 1993. Morgan Kaufmann.

Citation

MLA

Krogh, A., and J. Vedelsby. “Neural Network Ensembles, Cross Validation, and Active Learning”. Advances in Neural Information Processing Systems, vol. 7, 1994, https://proceedings.neurips.cc/paper_files/paper/1994/file/b8c37e33defde51cf91e1e03e51657da-Paper.pdf.

APA

Krogh, A., & Vedelsby, J. (1994). Neural Network Ensembles, Cross Validation, and Active Learning. Advances in Neural Information Processing Systems, 7. https://proceedings.neurips.cc/paper_files/paper/1994/file/b8c37e33defde51cf91e1e03e51657da-Paper.pdf

Chicago

Krogh, A., and J. Vedelsby. 1994. “Neural Network Ensembles, Cross Validation, and Active Learning”. Advances in Neural Information Processing Systems 7. https://proceedings.neurips.cc/paper_files/paper/1994/file/b8c37e33defde51cf91e1e03e51657da-Paper.pdf.

Harvard

Krogh, A. and Vedelsby, J. (1994) “Neural Network Ensembles, Cross Validation, and Active Learning”, Advances in Neural Information Processing Systems. Curran Associates, Inc. Available at: https://proceedings.neurips.cc/paper_files/paper/1994/file/b8c37e33defde51cf91e1e03e51657da-Paper.pdf.

Vancouver

1. Krogh A, Vedelsby J (1994) Neural Network Ensembles, Cross Validation, and Active Learning. Advances in Neural Information Processing Systems 7:

BibTeX

@inproceedings{krogh1994neural,
  title = {Neural Network Ensembles, Cross Validation, and Active Learning},
  author = {Krogh, Anders and Vedelsby, Jesper},
  year = {1994},
  booktitle = {Advances in Neural Information Processing Systems},
  publisher = {Curran Associates, Inc.},
  volume = {7},
  url = {https://proceedings.neurips.cc/paper_files/paper/1994/file/b8c37e33defde51cf91e1e03e51657da-Paper.pdf}
}

Metadata:DOI registry

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

Neural Network Ensembles, Cross Validation, and Active Learning

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Ensemble Ambiguity Decomposition of Generalization Error

Knowl 2 — Pointwise Ambiguity Identity and Ensemble Error Bounds

Knowl 3 — Quadratic Optimization of Ensemble Weights via Unlabeled Data

Knowl 4 — Optimality Condition for Non-Zero Ensemble Weights

Knowl 5 — Ambiguity-Based Active Learning for Regression

Knowl 6 — Cross-Validation Ensembling for Diversity and Error Estimation

Knowl 7 — Empirical Performance of Ambiguity-Based Active Learning

Knowl 8 — Empirical Verification of Cross-Validation Error Estimation and Weight Optimization

Knowl 9 — Limitations of Ambiguity as an Error Proxy and Cross-Validation Generalization Tradeoff

References

Citation

BibTeX

Access the Paper