Active Learning with Statistical Models

D. CohnZoubin GhahramaniMichael I. Jordan

article1996NeurIPS2,574 citations

Derives exact, closed-form criteria for active data selection through variance minimization in mixtures of Gaussians and locally weighted regression, enabling computationally efficient query selection that drastically reduces the training examples required for accurate regression.

Listen

The article addresses the challenge of efficiently acquiring training data for machine learning systems when each data point is costly or time-consuming to obtain, as is common in industrial or robotic applications. Passive approaches that rely on random or fixed data collection waste resources by including uninformative examples, whereas active learning lets the system select the most useful inputs to query. This matters now because many real-world tasks, such as robot control or sensor placement, involve high expenses per measurement and could benefit from drastic reductions in required data.

The article sets out to demonstrate that statistically optimal data selection, previously applied approximately and expensively to neural networks, can be performed efficiently and accurately with two alternative statistical models: mixtures of Gaussians and locally weighted regression. It derives closed-form criteria for choosing queries that minimize the learner's expected variance and tests them empirically.

The approach combines theoretical analysis with controlled experiments on a two-degree-of-freedom robot arm kinematics task. The models estimate output variance after a candidate query, integrate that expectation over a reference set drawn from the known input distribution, and select the query that yields the lowest average variance. Performance is compared against random sampling on learning curves for both estimated variance and mean squared error, using training sets up to several hundred examples and multiple runs with added input noise.

The key findings are that the variance-minimizing criterion produces learning curves with substantially lower error than random selection; for mixtures of Gaussians the mean squared error is consistently about one-third that of the random baseline, while locally weighted regression shows even larger gains and outperforms both mixtures and prior neural-network results. Computation times remain practical, scaling linearly with training-set size for mixtures and remaining fast enough for many control tasks with locally weighted regression. The optimality criterion sharply reduces the number of examples needed to reach good performance.

These results imply that organizations facing expensive data collection can achieve equivalent model accuracy with far fewer measurements, yielding major savings in time and cost while maintaining statistical grounding. The methods also produce parsimonious training sets that speed up subsequent predictions.

Next steps supported by the article include extending the framework to minimize bias as well as variance, deriving similar criteria for Bayesian networks and classification models, and testing on additional domains. Further work is needed to handle cases where the learner's bias is non-negligible and to incorporate regularization or convergence detection during fitting.

Limitations include the assumption that learners are approximately unbiased and that noise is roughly Gaussian; results are reported for a single low-dimensional task; and the input distribution must be known or approximable. Confidence is high for the tested conditions and models, but readers should be cautious about generalizing to high-bias settings or very high-dimensional problems without additional validation.

Harlequin Massachusetts Institute of Technology University of Toronto active learning correlation analysis feedforward neural networks Active Learning Statistical Model Mixture of Gaussians

arXiv: cs/9603104

Paper: Query by committee, H. Seung et al. (1992). This foundational paper establishes the query-by-committee paradigm for active learning using model disagreement, motivating the statistical and variance-reduction criteria developed in the source.
Paper: Neural Network Ensembles, Cross Validation, and Active Learning, Anders Krogh et al. (1994). It formulates active data selection based on ensemble ambiguity and variance, providing theoretical foundations for variance-minimizing query selection in statistical models.
Paper: Hierarchical Mixtures of Experts and the EM Algorithm, Michael I. Jordan et al. (1994). It introduces hierarchical mixtures of experts and their estimation via EM, establishing the mixture model architecture that the source adapts for closed-form active learning.
Paper: Adaptive Mixtures of Local Experts, Robert A. Jacobs et al. (1991). It presents the adaptive mixture of local experts framework used as a primary statistical architecture in the source's data selection experiments.

Paper: Combining active learning and semi-supervised learning using Gaussian fields and harmonic functions, Xiaojin Zhu et al. (2003). It extends variance- and risk-minimization active learning to graph-based semi-supervised settings using Gaussian random fields.
Paper: Support Vector Machine Active Learning with Applications to Text Classification, Simon Tong et al. (2001). It applies pool-based active learning principles to support vector machines by querying points to maximally reduce version space uncertainty.
Paper: Active Learning for Convolutional Neural Networks: A Core-Set Approach, Ozan Sener et al. (2018). It generalizes statistical active learning principles to deep convolutional neural networks via geometric core-set selection in batch settings.
Paper: A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning, Eric Brochu et al. (2010). It reviews Bayesian optimization techniques that build upon Gaussian process uncertainty quantification and variance reduction to actively query expensive black-box functions.
Paper: A Tutorial on Bayesian Optimization, Peter I. Frazier (2018). It details modern acquisition functions and Gaussian process surrogates for sequential sample selection, directly continuing the source's focus on statistical query optimization.

Cover for Active Learning with Statistical Models

Abstract

For many types of machine learning algorithms, one can compute the statistically “optimal” way to select training data. In this paper, we review how optimal data selection techniques have been used with feedforward neural networks. We then show how the same principles may be used to select data for two alternative, statistically-based learning architectures: mixtures of Gaussians and locally weighted regression. While the techniques for neural networks are computationally expensive and approximate, the techniques for mixtures of Gaussians and locally weighted regression are both efficient and accurate. Empirically, we observe that the optimality criterion sharply decreases the number of training examples the learner needs in order to achieve good performance.

Topics

active learning correlation analysis feedforward neural networks gaussians genetic selection

Keywords

Active Learning Statistical Model Mixture of Gaussians Locally Weighted Regression Variance Minimization Optimal Experiment Design EM Algorithm Neural Network Query Selection Bias Variance Tradeoff

1. Introduction
2. Active Learning – A Statistical Approach
2.1 Selecting Data to Minimize Learner Variance
2.2 Example: Active Learning with a Neural Network
3. Mixtures of Gaussians
3.1 Active Learning with a Mixture of Gaussians
4. Locally Weighted Regression
4.1 Setting the Smoothing Parameter k
4.2 Active Learning with Locally Weighted Regression
5. Experimental Results
5.1 Experiments with Mixtures of Gaussians
5.2 Experiments with LOESS Regression
5.3 Computation Time
6. Discussion
Appendix A. Notation
Acknowledgements
References

Knowls

Knowl 1 — Variance-Minimization Objective for Supervised Active Learning
theoretical result
In supervised active learning, the expected mean squared error of a learner's prediction $\hat{y}(x; \mathcal{D})$ given a training set $\mathcal{D}$ relative to the true target $y(x)$ , integrated over the marginal input distribution $P(x)$ , decomposes as:

$\int_X E_{\mathcal{T}}\left[(\hat{y}(x; \mathcal{D}) - y(x))^2 \mid x\right] P(x) \, dx = \int_X \left( E\left[(y(x) - E[y \mid x])^2\right] + (E_{\mathcal{D}}[\hat{y}(x; \mathcal{D})] - E[y \mid x])^2 + E_{\mathcal{D}}\left[(\hat{y}(x; \mathcal{D}) - E_{\mathcal{D}}[\hat{y}(x; \mathcal{D})])^2\right] \right) P(x) \, dx$

where $E_{\mathcal{T}}[\cdot]$ denotes expectation over $P(y \mid x)$ and training sets $\mathcal{D}$ , while $E_{\mathcal{D}}[\cdot]$ denotes expectation over training sets $\mathcal{D}$ with fixed input components. The first term is the noise variance of the output, the second is the learner's squared bias, and the third is the learner's prediction variance $\sigma_{\hat{y}}^2(x)$ .

Under the assumption that the learning model is approximately unbiased (squared bias is negligible compared to variance), active data selection minimizes expected future error by selecting a query input $\tilde{x}$ that minimizes the expected integrated learner variance:

$\int_X \langle \tilde{\sigma}_{\hat{y}}^2(x) \rangle P(x) \, dx$

where $\langle \tilde{\sigma}_{\hat{y}}^2(x) \rangle = E_{\mathcal{D} \cup \{(\tilde{x}, \tilde{y})\}}\left[ \sigma_{\hat{y} \mid \tilde{x}}^2(x) \right]$ is the expected variance at $x$ after incorporating query $(\tilde{x}, \tilde{y})$ , with expectation evaluated over the conditional output distribution $P(\tilde{y} \mid \tilde{x})$ .
Knowl 2 — Prediction and Output Variance in Gaussian Mixture Regression
model/method
A Mixture of Gaussians models the joint density on the input-output space $X \times Y$ using $N$ multivariate Gaussian components $g_i$ parameterized by means $\mu_i = [\mu_{x,i}, \mu_{y,i}]^T$ and covariances $\Sigma_i = \begin{bmatrix} \sigma_{x,i}^2 & \sigma_{xy,i} \\ \sigma_{xy,i} & \sigma_{y,i}^2 \end{bmatrix}$ .

For an arbitrary input $x$ , the responsibility $h_i(x)$ of component $g_i$ is:

$h_i(x) = \frac{P(x \mid i)}{\sum_{j=1}^N P(x \mid j)}, \quad \text{where } P(x \mid i) = \frac{1}{\sqrt{2\pi \sigma_{x,i}^2}} \exp\left( -\frac{(x - \mu_{x,i})^2}{2\sigma_{x,i}^2} \right)$

For each component $g_i$ , the conditional output expectation $\hat{y}_i(x)$ and conditional variance $\sigma_{\hat{y},i}^2(x)$ are:

$\hat{y}_i(x) = \mu_{y,i} + \frac{\sigma_{xy,i}}{\sigma_{x,i}^2}(x - \mu_{x,i}), \quad \sigma_{\hat{y},i}^2(x) = \frac{\sigma_{y \mid x,i}^2}{n_i} \left( 1 + \frac{(x - \mu_{x,i})^2}{\sigma_{x,i}^2} \right)$

where $\sigma_{y \mid x,i}^2 = \sigma_{y,i}^2 - \frac{\sigma_{xy,i}^2}{\sigma_{x,i}^2}$ is the conditional variance of $y$ given $x$ , and $n_i = \sum_{j=1}^m \frac{P(x_j, y_j \mid i)}{\sum_{k=1}^N P(x_j, y_j \mid k)}$ represents the training data support for Gaussian $g_i$ .

Assuming independence among the $\hat{y}_i(x)$ , the combined mixture output prediction and overall estimated model variance are:

$\hat{y}(x) = \sum_{i=1}^N h_i(x) \hat{y}_i(x), \quad \sigma_{\hat{y}}^2(x) = \sum_{i=1}^N \frac{h_i(x)^2 \sigma_{y \mid x,i}^2}{n_i} \left( 1 + \frac{(x - \mu_{x,i})^2}{\sigma_{x,i}^2} \right)$
Knowl 3 — Closed-Form Expected Variance Updates for Gaussian Mixture Active Learning
theoretical result
For a Mixture of Gaussians model, the estimated conditional distribution of an unobserved query label $\tilde{y}$ given query input $\tilde{x}$ is $P(\tilde{y} \mid \tilde{x}) = \sum_{i=1}^N \tilde{h}_i \mathcal{N}(\hat{y}_i(\tilde{x}), \sigma_{y \mid x,i}^2(\tilde{x}))$ , where $\tilde{h}_i = h_i(\tilde{x})$ . The expected new model variance at reference point $x$ resulting from querying candidate $\tilde{x}$ can be computed exactly in closed form as:

$\langle \tilde{\sigma}_{\hat{y}}^2(x) \rangle = \sum_{i=1}^N \frac{h_i(x)^2 \langle \tilde{\sigma}_{y \mid x,i}^2 \rangle}{n_i + \tilde{h}_i} \left( 1 + \frac{(x - \mu_{x,i})^2}{\sigma_{x,i}^2} \right)$

where the expected updated component parameters are:

$\langle \tilde{\sigma}_{y,i}^2 \rangle = \frac{n_i \sigma_{y,i}^2}{n_i + \tilde{h}_i} + \frac{n_i \tilde{h}_i \left( \sigma_{y \mid x,i}^2(\tilde{x}) + (\hat{y}_i(\tilde{x}) - \mu_{y,i})^2 \right)}{(n_i + \tilde{h}_i)^2}$

$\langle \tilde{\sigma}_{xy,i} \rangle = \frac{n_i \sigma_{xy,i}}{n_i + \tilde{h}_i} + \frac{n_i \tilde{h}_i (\tilde{x} - \mu_{x,i})(\hat{y}_i(\tilde{x}) - \mu_{y,i})}{(n_i + \tilde{h}_i)^2}$

$\langle \tilde{\sigma}_{xy,i}^2 \rangle = \langle \tilde{\sigma}_{xy,i} \rangle^2 + \frac{n_i^2 \tilde{h}_i^2 \sigma_{y \mid x,i}^2(\tilde{x}) (\tilde{x} - \mu_{x,i})^2}{(n_i + \tilde{h}_i)^4}$

$\langle \tilde{\sigma}_{y \mid x,i}^2 \rangle = \langle \tilde{\sigma}_{y,i}^2 \rangle - \frac{\langle \tilde{\sigma}_{xy,i}^2 \rangle}{\sigma_{x,i}^2}$

When $\mu_{x,i}$ and $\sigma_{x,i}^2$ are also estimated from data, their updated expected values $\tilde{\mu}_{x,i} = \frac{n_i \mu_{x,i} + \tilde{h}_i \tilde{x}}{n_i + \tilde{h}_i}$ and $\tilde{\sigma}_{x,i}^2 = \frac{n_i \sigma_{x,i}^2}{n_i + \tilde{h}_i} + \frac{n_i \tilde{h}_i (\tilde{x} - \mu_{x,i})^2}{(n_i + \tilde{h}_i)^2}$ substitute into the variance equations.
Knowl 4 — Input Distribution Correction for Gaussian Mixture Active Learning
model/method
Because active data selection generates a training input distribution $P(\tilde{x})$ that departs from the true input distribution $P(x)$ , estimating parameters directly from the active training data produces the conditioned distribution $P(\tilde{x}, y \mid i)$ rather than $P(x, y \mid i)$ . To correct for this sampling bias, the joint density estimate for Gaussian component $i$ is weighted by the ratio of the marginal input densities:

$P(x, y \mid i) = P(\tilde{x}, y \mid i) \frac{P(x \mid i)}{P(\tilde{x} \mid i)}$

where $P(\tilde{x} \mid i)$ is computed using the mean and variance of component $i$ on the training data, and $P(x \mid i)$ is computed using the mean and variance of component $i$ evaluated on an unlabelled reference dataset sampled from the known marginal input distribution $P(x)$ .
Knowl 5 — Prediction and Output Variance Formulation for Locally Weighted Regression
model/method
In locally weighted regression (LOESS), predictions around a point $x$ are obtained by fitting a weighted linear regression over training samples $(x_i, y_i)_{i=1}^m$ using kernel weights $h_i(x) = \exp(-k(x - x_i)^2)$ , where $k$ is a smoothing parameter. Setting total weight $n = \sum_i h_i(x)$ , the kernel-weighted means and covariances are:

$\mu_x = \frac{\sum_i h_i x_i}{n}, \quad \mu_y = \frac{\sum_i h_i y_i}{n}$

$\sigma_x^2 = \frac{\sum_i h_i (x_i - \mu_x)^2}{n}, \quad \sigma_y^2 = \frac{\sum_i h_i (y_i - \mu_y)^2}{n}, \quad \sigma_{xy} = \frac{\sum_i h_i (x_i - \mu_x)(y_i - \mu_y)}{n}$

$\sigma_{y \mid x}^2 = \sigma_y^2 - \frac{\sigma_{xy}^2}{\sigma_x^2}$

The model prediction $\hat{y}(x)$ and estimated prediction variance $\sigma_{\hat{y}}^2(x)$ are given in closed form by:

$\hat{y}(x) = \mu_y + \frac{\sigma_{xy}}{\sigma_x^2}(x - \mu_x)$

$\sigma_{\hat{y}}^2(x) = \frac{\sigma_{y \mid x}^2}{n^2} \left( \sum_i h_i^2 + \frac{(x - \mu_x)^2}{\sigma_x^2} \sum_i h_i^2 \frac{(x_i - \mu_x)^2}{\sigma_x^2} \right)$
Knowl 6 — Expected Variance Updates and Fast Moment Caching in LOESS Active Learning
theoretical result
For a LOESS model with kernel weight $\tilde{h} = h(\tilde{x} - x)$ assigned to a proposed candidate query $\tilde{x}$ , the expected new output variance $\langle \tilde{\sigma}_{\hat{y}}^2(x) \rangle$ at reference point $x$ is:

$\langle \tilde{\sigma}_{\hat{y}}^2(x) \rangle = \frac{\langle \tilde{\sigma}_{y \mid x}^2 \rangle}{(n + \tilde{h})^2} \left[ \sum_i h_i^2 + \tilde{h}^2 + \frac{(x - \tilde{\mu}_x)^2}{\tilde{\sigma}_x^2} \left( \sum_i h_i^2 \frac{(x_i - \tilde{\mu}_x)^2}{\tilde{\sigma}_x^2} + \tilde{h}^2 \frac{(\tilde{x} - \tilde{\mu}_x)^2}{\tilde{\sigma}_x^2} \right) \right]$

where updated parameters and expectations are:

$\tilde{\mu}_x = \frac{n \mu_x + \tilde{h}\tilde{x}}{n + \tilde{h}}, \quad \tilde{\sigma}_x^2 = \frac{n \sigma_x^2}{n + \tilde{h}} + \frac{n \tilde{h}(\tilde{x} - \mu_x)^2}{(n + \tilde{h})^2}$

$\langle \tilde{\sigma}_y^2 \rangle = \frac{n \sigma_y^2}{n + \tilde{h}} + \frac{n \tilde{h} \left( \sigma_{y \mid \tilde{x}}^2 + (\hat{y}(\tilde{x}) - \mu_y)^2 \right)}{(n + \tilde{h})^2}$

$\langle \tilde{\sigma}_{xy} \rangle = \frac{n \sigma_{xy}}{n + \tilde{h}} + \frac{n \tilde{h}(\tilde{x} - \mu_x)(\hat{y}(\tilde{x}) - \mu_y)}{(n + \tilde{h})^2}$

$\langle \tilde{\sigma}_{xy}^2 \rangle = \langle \tilde{\sigma}_{xy} \rangle^2 + \frac{n^2 \tilde{h}^2 \sigma_{y \mid \tilde{x}}^2 (\tilde{x} - \mu_x)^2}{(n + \tilde{h})^4}, \quad \langle \tilde{\sigma}_{y \mid x}^2 \rangle = \langle \tilde{\sigma}_y^2 \rangle - \frac{\langle \tilde{\sigma}_{xy}^2 \rangle}{\tilde{\sigma}_x^2}$

To avoid recomputing the sum $\sum_i h_i^2 (x_i - \tilde{\mu}_x)^2$ in $O(m)$ time for each candidate, the term is expanded as $\sum_i h_i^2 x_i^2 + \tilde{\mu}_x^2 \sum_i h_i^2 - 2\tilde{\mu}_x \sum_i h_i^2 x_i$ , allowing evaluation in $O(1)$ time by caching $\sum_i h_i^2 x_i^2$ , $\sum_i h_i^2 x_i$ , and $\sum_i h_i^2$ .
Knowl 7 — Variance-Minimizing Kernel Smoothing Parameter Selection in LOESS
model/method
In locally weighted regression with kernel $h_i(x) = \exp(-k(x - x_i)^2)$ , the smoothing parameter $k$ controls estimator locality. As $k$ decreases, the regression incorporates more global data, increasing total weight $n$ (which reduces variance $\sigma_{\hat{y}}^2$ ) but increasing the local conditional variance $\sigma_{y \mid x}^2$ due to nonlinear deviations (which increases variance $\sigma_{\hat{y}}^2$ ).

The variance-minimizing selection criterion chooses $k$ to directly minimize estimated output variance $\sigma_{\hat{y}}^2$ over a set of reference points drawn from the input distribution $P(x)$ . This balances effective support against local bias without requiring assumptions about noise levels or cross-validation on an unrepresentative active training set distribution.
Knowl 8 — Variance-Minimizing Active Query Selection Algorithm
algorithm
Input: Input distribution $P(x)$ , initial dataset $\mathcal{D}_0$ , reference sample size $R = 64$ , candidate sample size $C = 64$ , iterations $T$ .
Output: Augmented dataset $\mathcal{D}_T$ and trained statistical model.
Initialize $\mathcal{D} \leftarrow \mathcal{D}_0$
Fit model (Mixture of Gaussians via EM or LOESS via local weighted sums) on $\mathcal{D}$
for $t = 1$ to $T$ do
    Sample $R$ reference points $\{x_r\}_{r=1}^R$ from $P(x)$
    Sample $C$ candidate query points $\{\tilde{x}_c\}_{c=1}^C$ uniformly from input domain
    for each candidate $\tilde{x}_c$ do
         $V(\tilde{x}_c) \leftarrow \frac{1}{R} \sum_{r=1}^R \langle \tilde{\sigma}_{\hat{y}}^2(x_r) \rangle$
    end for
    Select optimal query $\tilde{x}^* \leftarrow \arg\min_{\tilde{x}_c} V(\tilde{x}_c)$
    Query oracle to observe output $\tilde{y}^*$ at $\tilde{x}^*$
    Update dataset $\mathcal{D} \leftarrow \mathcal{D} \cup \{(\tilde{x}^*, \tilde{y}^*)\}$
    Retrain or update model parameters on $\mathcal{D}$
end for
return $\mathcal{D}$ and trained model
The algorithm iteratively draws an independent reference set to represent $P(x)$ , scores candidate queries by the Monte Carlo average of the closed-form expected variance $\langle \tilde{\sigma}_{\hat{y}}^2 \rangle$ , selects the variance-minimizing point, and updates the model.
Knowl 9 — Empirical Generalization on Robot Arm Kinematics Benchmark
empirical result
On the 2-degree-of-freedom robot arm kinematics domain (Arm2D) where inputs are joint angles $(\theta_1, \theta_2)$ , outputs are tip coordinates $(X_1, X_2)$ , and $1\%$ additive Gaussian noise is added to the inputs (inducing non-Gaussian output error):
- For a Mixture of 60 Gaussians (trained with 20 EM iterations per step), the variance-minimizing active learning criterion produces a mean squared error (MSE) on independent test points that is consistently $1/3$ of the MSE achieved by random data selection across training set sizes up to 500 points.
- For LOESS regression with Gaussian kernels, both estimated output variance and MSE under variance-minimizing query selection are significantly lower than under random sampling.
- LOESS regression significantly outperforms both the Mixture of Gaussians and feedforward neural networks in generalization accuracy and data efficiency on the Arm2D task.
Knowl 10 — Computational Time Complexity of Statistical Active Learning
data/table
Could not parse LaTeX table
Computation times measured on a Sun Sparc 10 workstation as a function of training set size $m$ (Mixture of Gaussians used 60 components trained for 20 iterations; reference times are per reference point, candidate times are per candidate point per reference point).

For the Mixture of Gaussians, training time scales linearly with $m$ , but candidate evaluation per reference point is constant ( $1300\,\mu\text{s}$ ) regardless of training set size. For LOESS, there is no explicit training phase; candidate evaluation scales with training size as $58 + 0.16m\,\mu\text{s}$ per candidate per reference point. For a setup with 100 training points, 64 reference points, and 64 candidate points, LOESS selects an optimal query in approximately 0.3 seconds.

Coverage note — The paper's background review of Optimal Experiment Design applied to feedforward neural networks (Fedorov 1972; MacKay 1992; Cohn 1994) was omitted from the main knowls because it represents existing literature rather than this paper's new contributions.

References

1.Angluin, D. (1988). Queries and concept learning. Machine Learning, 2, 319–342.
2.Baum, E., & Lang, K. (1991). Neural network algorithms that learn in polynomial time from examples and queries. IEEE Trans. Neural Networks, 2.
3.Box, G., & Draper, N. (1987). Empirical model-building and response surfaces. Wiley.
4.Cheeseman, P., Self, M., Kelly, J., Taylor, W., Freeman, D., & Stutz, J. (1988). Bayesian classification. In AAAI 88, The 7th National Conference on Artificial Intelligence, pp. 607–611. AAAI Press.
5.Cleveland, W., Devlin, S., & Grosse, E. (1988). Regression by local fitting. Journal of Econometrics, 37, 87–114.
6.Cohn, D. (1994). Neural network exploration using optimal experiment design. In Cowan, J., Tesauro, G., & Alspector, J. (Eds.), Advances in Neural Information Processing Systems 6. Morgan Kaufmann. Expanded version available as MIT AI Lab memo 1491 by anonymous ftp to publications.ai.mit.edu.
7.Cohn, D. (1995). Minimizing statistical bias with queries. AI Lab memo AIM-1552, Massachusetts Institute of Technology. Available by anonymous ftp from publications.ai.mit.edu.
8.Cohn, D., Atlas, L., & Ladner, R. (1990). Training connectionist networks with queries and selective sampling. In Touretzky, D. (Ed.), Advances in Neural Information Processing Systems 2. Morgan Kaufmann.
9.Cohn, D., Atlas, L., & Ladner, R. (1994). Improving generalization with active learning. Machine Learning, 5 (2), 201–221.
10.Dempster, A., Laird, N., & Rubin, D. (1977). Maximum likelihood from incomplete data via the EM algorithm. J. Royal Statistical Society Series B, 39, 1–38.
11.Fedorov, V. (1972). Theory of Optimal Experiments. Academic Press.
12.Fe'ldbaum, A. A. (1965). Optimal control systems. Academic Press, New York, NY.
13.Geman, S., Bienenstock, E., & Doursat, R. (1992). Neural networks and the bias/variance dilemma. Neural Computation, 4, 1–58.
14.Ghahramani, Z., & Jordan, M. (1994). Supervised learning from incomplete data via an EM approach. In Cowan, J., Tesauro, G., & Alspector, J. (Eds.), Advances in Neural Information Processing Systems 6. Morgan Kaufmann.
15.Heckerman, D., Geiger, D., & Chickering, D. (1994). Learning Bayesian networks: the combination of knowledge and statistical data. Tech report MSR-TR-94-09, Microsoft.
16.Linden, A., & Weber, F. (1993). Implementing inner drive by competence reflection. In Roitblat, H. (Ed.), Proceedings of the 2nd International Conference on Simulation of Adaptive Behavior. MIT Press, Cambridge, MA.
17.MacKay, D. J. (1992). Information-based objective functions for active data selection. Neural Computation, 4 (4), 590–604.
18.Nowlan, S. (1991). Soft competitive adaptation: Neural network learning algorithms based on fitting statistical mixtures. Tech report CS-91-126, Carnegie Mellon University.
19.Paass, G., & Kindermann, J. (1995). Bayesian query construction for neural network models. In Tesauro, G., Touretzky, D., & Leen, T. (Eds.), Advances in Neural Information Processing Systems 7. MIT Press.
20.Pearl, J. (1988). Probablistic Reasoning in Intelligent Systems. Morgan Kaufmann.
21.Plutowski, M., & White, H. (1993). Selecting concise training sets from clean data. IEEE Transactions on Neural Networks, 4, 305–318.
22.Schaal, S., & Atkeson, C. (1994). Robot juggling: An implementation of memory-based learning. Control Systems, 14, 57–71.
23.Schmidhuber, J., & Storck, J. (1993). Reinforcement driven information acquisition in non-deterministic environments. Tech report, Fakultät für Informatik, Technische Universität München.
24.Specht, D. (1991). A general regression neural network. IEEE Trans. Neural Networks, 2 (6), 568–576.
25.Thrun, S., & Möller, K. (1992). Active exploration in dynamic environments. In Moody, J., Hanson, S., & Lippmann, R. (Eds.), Advances in Neural Information Processing Systems 4. Morgan Kaufmann.
26.Titterington, D., Smith, A., & Makov, U. (1985). Statistical Analysis of Finite Mixture Distributions. Wiley.
27.Weisberg, S. (1985). Applied Linear Regression. Wiley.
28.Whitehead, S. (1991). A study of cooperative mechanisms for faster reinforcement learning. Technical report CS-365, University of Rochester, Rochester, NY.

Citation

MLA

Cohn, D. A., et al. “Active Learning with Statistical Models”. Journal of Artificial Intelligence Research, vol. 4, 1996, pp. 129–45, https://doi.org/10.1613/jair.295.

APA

Cohn, D. A., Ghahramani, Z., & Jordan, M. I. (1996). Active Learning with Statistical Models. Journal of Artificial Intelligence Research, 4, 129–145. https://doi.org/10.1613/jair.295

Chicago

Cohn, D. A., Z. Ghahramani, and M. I. Jordan. 1996. “Active Learning with Statistical Models”. Journal of Artificial Intelligence Research 4: 129–45. https://doi.org/10.1613/jair.295.

Harvard

Cohn, D.A., Ghahramani, Z. and Jordan, M.I. (1996) “Active Learning with Statistical Models”, Journal of Artificial Intelligence Research, 4, pp. 129–145. Available at: https://doi.org/10.1613/jair.295.

Vancouver

1. Cohn DA, Ghahramani Z, Jordan MI (1996) Active Learning with Statistical Models. Journal of Artificial Intelligence Research 4:129–145

BibTeX

@article{Cohn_1996, title={Active Learning with Statistical Models}, volume={4}, ISSN={1076-9757}, url={http://dx.doi.org/10.1613/jair.295}, DOI={10.1613/jair.295}, journal={Journal of Artificial Intelligence Research}, publisher={AI Access Foundation}, author={Cohn, D. A. and Ghahramani, Z. and Jordan, M. I.}, year={1996}, month=Mar, pages={129–145} }

Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

Active Learning with Statistical Models

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Variance-Minimization Objective for Supervised Active Learning

Knowl 2 — Prediction and Output Variance in Gaussian Mixture Regression

Knowl 3 — Closed-Form Expected Variance Updates for Gaussian Mixture Active Learning

Knowl 4 — Input Distribution Correction for Gaussian Mixture Active Learning

Knowl 5 — Prediction and Output Variance Formulation for Locally Weighted Regression

Knowl 6 — Expected Variance Updates and Fast Moment Caching in LOESS Active Learning

Knowl 7 — Variance-Minimizing Kernel Smoothing Parameter Selection in LOESS

Knowl 8 — Variance-Minimizing Active Query Selection Algorithm

Knowl 9 — Empirical Generalization on Robot Arm Kinematics Benchmark

Knowl 10 — Computational Time Complexity of Statistical Active Learning

References

Citation

BibTeX

Access the Paper