Robust Classification for Imprecise Environments

F. ProvostTom Fawcett

article2000Machine-mediated learning1,384 citations

Introduces the ROC convex hull method to construct hybrid classifiers that guarantee optimal performance across changing misclassification costs and class distributions without committing to a single model in advance.

Listen

Real-world decision-support and predictive systems—such as those used for fraud detection, medical screening, and equipment fault diagnosis—frequently operate under conditions where true misclassification costs and target class distributions are uncertain or constantly shifting. Traditional model evaluation approaches select a single "best" model based on fixed assumptions, such as overall accuracy or a predetermined cost ratio. This standard approach produces brittle systems that quickly become suboptimal or ineffective as real-world operational conditions change.

The article demonstrates a robust method for evaluating and deploying classification models in imprecise environments. Its objective is to show that decision-makers can avoid premature commitment to a single model by constructing an optimal hybrid system that performs at least as well as, and sometimes better than, any individual candidate model under any operating condition.

The researchers combine concepts from signal detection theory, decision analysis, and computational geometry to create the Receiver Operating Characteristic Convex Hull (ROCCH) method. They map the false alarm and hit rates of candidate models into visual trade-off curves, identifying the convex hull—the outer boundary representing the highest possible performance across all potential cost and distribution trade-offs. To validate their approach, the authors analyzed empirical data from six clinical datasets evaluated by Andrew Bradley and ten benchmark datasets from the University of California, Irvine (UCI) repository, testing diverse algorithms including decision trees, Naive Bayes, and nearest-neighbor models across multiple cross-validation runs.

The investigation produced four central findings. First, single universally best models are exceptionally rare; across all six clinical datasets and in nine out of ten UCI benchmark datasets, no individual model dominated across all operational trade-offs. Second, the ROCCH-hybrid systematically matches the lowest possible expected cost across any target environment, achieving optimal performance for standard operational metrics such as accuracy, lift, precision, recall, and workforce capacity constraints. Third, by probabilistically interpolating between adjacent models along the outer hull boundary, the hybrid system can achieve specific false-positive-to-true-positive trade-offs that exceed the performance of any single standalone model. Fourth, the method is computationally efficient to build and store, scaling efficiently at O(N log N) time for N models and discarding all suboptimal models that fail to touch the convex boundary.

These findings indicate that choosing a single model during the system-building phase introduces substantial risk and hidden operational costs. The ROCCH-hybrid provides significant strategic value by deferring final operating choices to run time. When conditions shift—such as seasonal variations in fraud rates or changing labor capacities to investigate flagged incidents—system operators can adjust system sensitivity using a single operational control without retraining or rebuilding models from scratch.

Organizations developing classification systems should replace single-model selection routines with the ROCCH evaluation framework. Teams should maintain the portfolio of hull-defining models and tune operating thresholds at deployment based on real-time feedback and business constraints. The analysis focuses primarily on two-class classification problems with uniform costs per error type and assumes reliable initial sample estimates. Where decisions involve multi-class outcomes or highly variable instance-level costs, technical teams should conduct focused pilot evaluations to calibrate and validate local curve estimates before deployment.

Hewlett-Packard Labs New York University classification models cross validation decision analysis robust classification ROCCH method ROC analysis

arXiv: cs/0009007

Paper: MetaCost: a general method for making classifiers cost-sensitive, Pedro M. Domingos (1999). Introduces foundational cost-sensitive learning principles and cost matrices that motivate the need for classifier evaluation under imprecise operating conditions.
Paper: A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection, Ron Kohavi (1995). Establishes the cross-validation and resampling protocols used in the source paper to reliably assess candidate classification models across benchmark datasets.
Paper: Addressing the Curse of Imbalanced Training Sets: One-Sided Selection, Miroslav Kubát et al. (1997). Examines the severe performance degradation of standard classifiers under class-skewed distributions that the ROC convex hull method is designed to address.
Paper: Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms, Thomas G. Dietterich (1998). Provides the essential statistical testing frameworks required to evaluate whether differences among supervised classification algorithms are statistically meaningful.
Paper: Neural Network Ensembles, Lars Kai Hansen et al. (1990). Demonstrates early foundational principles for combining multiple candidate models to achieve superior decision boundaries compared to any single standalone classifier.

Paper: The foundations of cost-sensitive learning, Charles Elkan (2001). Formalizes the theoretical foundations and conditions of cost-sensitive learning and threshold adjustments across shifting base rates and misclassification costs.
Paper: A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems, DAVID J. HAND et al. (2001). Generalizes cost-independent ROC analysis to multi-class classification problems, directly extending two-class evaluation methods.
Paper: Using AUC and accuracy in evaluating learning algorithms, Jin Huang et al. (2005). Builds upon ROC analysis by formally proving the statistical consistency and discriminative superiority of AUC over conventional accuracy metrics.
Paper: The relationship between Precision-Recall and ROC curves, Jesse Davis et al. (2006). Extends the geometric analysis of ROC space by establishing the formal mathematical relationship and convex hull equivalences with Precision-Recall space under heavy class skew.
Paper: Predicting good probabilities with supervised learning, Alexandru Niculescu-Mizil et al. (2005). Investigates post-processing probability calibration techniques that refine the probability estimates required when choosing optimal thresholds along the ROC convex hull.
Paper: SMOTE: Synthetic Minority Over-sampling Technique, N. Chawla et al. (2002). Applies ROC convex hull analysis to evaluate synthetic sampling strategies specifically designed to overcome extreme class imbalance.
Paper: Mining concept-drifting data streams using ensemble classifiers, Haixun Wang et al. (2003). Extends robust ensemble evaluation to dynamic data streams experiencing continuous concept drift and shifting operational conditions.

Cover for Robust Classification for Imprecise Environments

Abstract

In real-world environments it usually is difficult to specify target operating conditions precisely, for example, target misclassification costs. This uncertainty makes building robust classification systems problematic. We show that it is possible to build a hybrid classifier that will perform at least as well as the best available classifier for any target conditions. In some cases, the performance of the hybrid actually can surpass that of the best known classifier. This robust performance extends across a wide variety of comparison frameworks, including the optimization of metrics such as accuracy, expected cost, lift, precision, recall, and workforce utilization. The hybrid also is efficient to build, to store, and to update. The hybrid is based on a method for the comparison of classifier performance that is robust to imprecise class distributions and misclassification costs. The ROC convex hull (ROCCH) method combines techniques from ROC analysis, decision analysis and computational geometry, and adapts them to the particulars of analyzing learned classifiers. The method is efficient and incremental, minimizes the management of classifier performance data, and allows for clear visual comparisons and sensitivity analyses. Finally, we point to empirical evidence that a robust hybrid classifier indeed is needed for many real-world problems.

Topics

classification models cross validation decision analysis genetic selection geometry k-nearest neighbors mode selection naive bayes signal detection theory trade

Keywords

robust classification ROCCH method ROC analysis cost-sensitive learning misclassification costs class priors decision trees hybrid classifier

1 Introduction
1.1 An example
1.2 Basic terminology
2 Evaluating and visualizing classifier performance
2.1 Classifier comparison: decision analysis and ROC analysis
2.2 The ROC Convex Hull method
2.2.1 Iso-performance lines
2.2.2 The ROC convex hull
2.2.3 Generating the ROC Convex Hull
2.2.4 Comparing a variety of classifiers
2.2.5 Changing distributions and costs
2.2.6 Sensitivity analysis
3 Building robust classifiers
3.1 ROCCH-hybrid classifiers
3.2 Robust classification
3.2.1 Minimizing expected cost
3.3 Robust classification for other common metrics
3.4 Ranking cases
3.5 Whole-curve metrics
3.6 Using the ROCCH-hybrid
3.7 Beating the component classifiers
3.8 Time and space efficiency
4 Empirical demonstration of need
4.1 Bradley’s study
4.2 Our study
5 Limitations and future work
6 Conclusion
7 Acknowledgments
References

Knowls

Knowl 1 — Iso-Performance Lines in ROC Space
equation
In binary classification with positive class $p$ and negative class $n$ , let $TP = p(Y \mid p)$ denote the true positive rate (hit rate) and $FP = p(Y \mid n)$ denote the false positive rate (false alarm rate), where $Y$ represents a positive prediction. Let $c(Y, n)$ be the cost of a false positive error, $c(N, p)$ be the cost of a false negative error (with zero cost for correct predictions), and let $p(p)$ and $p(n) = 1 - p(p)$ denote the prior probabilities of the positive and negative classes, respectively.

The expected cost $ec(FP, TP)$ of a classifier represented by point $(FP, TP)$ in ROC space is: $ec(FP, TP) = p(p) \cdot (1 - TP) \cdot c(N, p) + p(n) \cdot FP \cdot c(Y, n)$

Two points $(FP_1, TP_1)$ and $(FP_2, TP_2)$ achieve identical expected cost if they lie on a line with slope $m$ : $m = \frac{c(Y, n) \, p(n)}{c(N, p) \, p(p)}$

This defines an iso-performance line. All classifiers lying along the same iso-performance line yield equal expected cost. Families of parallel lines positioned further northwest (possessing higher $TP$ -axis intercepts) represent strictly lower expected cost.
Knowl 2 — ROC Convex Hull Selection of Potentially Optimal Classifiers
model/method
The ROC convex hull (ROCCH) of a set of classifiers is the convex hull of their corresponding operating points $(FP, TP)$ in ROC space. The northwest boundary of this hull (the upper boundary lying above the uninformative diagonal line $y = x$ ) contains the complete set of classifiers that are potentially optimal under any possible class distribution and cost assignment.

For any target operating environment defining an iso-performance line slope $m = \frac{c(Y, n)p(n)}{c(N, p)p(p)} \ge 0$ , the optimal classifier is the point on the ROC convex hull that intersects the iso-performance line with the largest $TP$ -intercept. Any classifier whose operating point lies strictly in the interior of the convex hull or below its northwest boundary will have a higher expected cost than at least one point on the hull for all non-negative cost and probability distributions, and can be discarded without loss of potential optimality.
Knowl 3 — ROCCH-Hybrid Classifier Construction and Interpolation
model/method
An ROCCH-hybrid classifier is a composite classifier $\mu(I, x, C)$ that selects or interpolates among the set $C$ of classifiers forming the vertices of the ROC convex hull to realize any operating point $(x, TP_x)$ on the hull boundary.

For a target false positive rate $x \in [0, 1]$ and an instance $I$ :
1. If $(x, TP_x)$ is a vertex of the ROC convex hull, the hybrid evaluates $I$ directly using the classifier associated with that vertex.
2. If $(x, TP_x)$ is a non-vertex point lying on a hull segment between adjacent vertices $C_l = (FP_l, TP_l)$ and $C_r = (FP_r, TP_r)$ where $FP_l < x < FP_r$ , the hybrid interpolates between $C_l$ and $C_r$ using a randomized decision rule. For each input instance $I$ , the hybrid emits the classification produced by $C_r$ with probability $\frac{x - FP_l}{FP_r - FP_l}$ , and emits the classification produced by $C_l$ with probability $1 - \frac{x - FP_l}{FP_r - FP_l}$ .
The resulting hybrid classifier attains an expected false positive rate of exactly $x$ and an expected true positive rate of $TP_x$ , allowing the system to achieve operating points strictly superior to any individual base classifier.
Knowl 4 — Optimality of ROCCH-Hybrid for Expected Cost and Accuracy
theoretical result
For any two-class problem with arbitrary non-negative misclassification costs $c(Y, n)$ and $c(N, p)$ and arbitrary class priors $p(p)$ and $p(n) = 1 - p(p)$ , the ROCCH-hybrid classifier achieves the minimum expected cost among all available classifiers and their combinations.

Because the ROC convex hull is a piecewise-linear, concave-down curve, its segment slopes are monotonically non-increasing. For any iso-performance slope $m_{ec} = \frac{c(Y, n)p(n)}{c(N, p)p(p)} \ge 0$ , the point where the slope of the ROC convex hull equals $m_{ec}$ minimizes the expected cost function $ec(FP, TP)$ .

Under uniform misclassification costs ( $c(Y, n) = c(N, p)$ ), minimizing expected cost is equivalent to minimizing undifferentiated error rate. Consequently, the ROCCH-hybrid simultaneously maximizes classification accuracy for any target class distribution $p(p)$ .
Knowl 5 — Optimality of ROCCH-Hybrid under Constrained Monotonic Metrics
theoretical result
Let $f(FP, TP)$ be any classifier evaluation metric that is monotonically non-decreasing in true positive rate and monotonically non-increasing in false positive rate: $\frac{\partial f}{\partial TP} \ge 0 \quad \text{and} \quad \frac{\partial f}{\partial FP} \le 0$

Under any single linear constraint on classifier performance of the form: $a \cdot TP + b \cdot FP \le c$ where $a, b \ge 0$ and $c \in \mathbb{R}$ , the ROCCH-hybrid achieves a metric value $f(FP, TP)$ at least as high as that of any constituent classifier.

This optimality guarantee applies directly to several common operational settings:
1. Neyman-Pearson Decision Criterion: Maximizing $TP$ subject to $FP \le FP_{\max}$ .
2. Workforce Utilization (Fixed Capacity): Maximizing true positives when a fixed budget of $K$ total cases can be reviewed, defined by the constraint $TP \cdot P + FP \cdot N \le K$ , where $P$ and $N$ are the total positive and negative instances.
3. Information Retrieval and Marketing Metrics: Maximizing precision, recall, or lift subject to percentage or absolute case presentation cutoffs.
Knowl 6 — Single-Pass Generation of ROC Curves from Ranked Examples
algorithm
This algorithm constructs an ROC curve from a set of test instances scored by a continuous-output classification model. By delaying the addition of curve vertices until all instances sharing the same score have been tallied, the algorithm eliminates ordering bias among score ties.
Input: A list $E$ of tuples $\langle I, p \rangle$ , where $I$ is an instance with true class in $\{p, n\}$ and $p$ is its predicted numeric score; $P$ , the total count of positive examples in $E$ ; $N$ , the total count of negative examples in $E$ .
Output: $R$ , an ordered list of $(FP, TP)$ coordinates defining the ROC curve.
$Tcount \leftarrow 0$
$Fcount \leftarrow 0$
$plast \leftarrow -\infty$
$R \leftarrow \langle \rangle$
Sort $E$ in decreasing order by score $p$
while $E \neq \emptyset$ do
    Remove tuple $\langle I, p \rangle$ from the head of $E$
    if $p \neq plast$ then
        Append $(Fcount / N, Tcount / P)$ to the end of $R$
         $plast \leftarrow p$
    end if
    if $I$ is a positive example then
         $Tcount \leftarrow Tcount + 1$
    else
         $Fcount \leftarrow Fcount + 1$
    end if
end while
Append $(Fcount / N, Tcount / P)$ to the end of $R$
return $R$
Knowl 7 — AUC Maximization of the ROCCH-Hybrid
theoretical result
The Area Under the ROC Curve (AUC) corresponds to the probability that a classifier ranks a randomly chosen positive instance higher than a randomly chosen negative instance. Because the ROC convex hull is the upper convex envelope enclosing all base ROC curves and discrete classifier points in ROC space, the area under the ROC convex hull is strictly greater than or equal to the AUC of every individual base classifier or linear combination thereof. Thus, the ROCCH-hybrid achieves the maximal achievable AUC over the available pool of models.
Knowl 8 — Empirical Multi-Classifier Dominance Across Benchmark Domains
empirical result
Empirical evaluation reveals that globally dominating classifiers almost never occur in practical settings:
1. Across six medical datasets evaluated with six distinct learning algorithms (two neural networks, two decision trees, and two statistical methods), zero datasets contained a single classifier that dominated all others across the entire ROC space.
2. Across ten UCI repository datasets evaluated with 10-fold cross-validation using MC4 decision trees, Naive Bayes with discretization, $k$ -nearest neighbors (IB1, IB3, IB5), and Bagged-MC4, only one domain (Vehicle) exhibited an absolute dominating classifier across all cross-validation folds.
Because different models are optimal across different segments of ROC space, selecting a single model prior to deployment leads to suboptimal performance when operating conditions or costs deviate from experimental assumptions.

Knowl 9 — Locally Dominating Classifiers Across Iso-Performance Slope Ranges in Benchmark Domains

data/table

The table below details the component classifiers forming the ROC convex hull across varying iso-performance slope ranges $m = \frac{c(Y, n)p(n)}{c(N, p)p(p)}$ across four UCI benchmark datasets.

Domain	Slope range	Dominator
Vehicle	$[0, \infty)$	Bagged-MC4
Road (Grass)	$[0, 0.38]$	Naive Bayes (NB)
	$[0.38, \infty)$	Bagged-MC4
CRX	$[0, 0.03]$	Bagged-MC4
	$[0.03, 0.06]$	Naive Bayes (NB)
	$[0.06, 2.06]$	Bagged-MC4
	$[2.06, \infty)$	Naive Bayes (NB)
Satimage	$[0, 0.05]$	Naive Bayes (NB)
	$[0.05, 0.22]$	Bagged-MC4
	$[0.22, 2.60]$	IB5 ( $5$ -NN)
	$[2.60, 3.11]$	IB3 ( $3$ -NN)
	$[3.11, 7.54]$	IB5 ( $5$ -NN)
	$[7.54, 31.14]$	IB3 ( $3$ -NN)
	$[31.14, \infty)$	Bagged-MC4

In complex domains such as Satimage, four distinct learning algorithms alternate in optimality across seven distinct slope ranges, showing that no single model can optimize performance over the full range of cost and class distributions.

Knowl 10 — Limitations and Multi-Class Complexity of the ROCCH Method
limitation
The ROCCH method possesses several operational and theoretical limitations:
1. Multi-Class Dimensionality: In a problem with $C$ classes, the ROC space generalizes to an ROC hyperspace with dimension $C(C - 1)$ . As $C$ increases, the computational complexity of computing the convex hull grows exponentially with dimension, and direct visualization becomes infeasible.
2. Assumption of Constant Intra-Type Costs: The standard ROCCH assumes uniform costs for all errors of the same category (e.g., all false positives incur equal cost). For problems with example-dependent costs, instances must be resampled or weighted proportionally to their individual costs.
3. Sample Estimation Variance: Constructing reliable confidence bands over empirical ROC curves and convex hulls remains non-trivial due to variance in finite test sample estimates.
4. Model Comprehensibility: While individual component models may be interpretable, interpolating between two distinct models via randomized predictions impairs the overall comprehensibility and explanatory power of the resulting classifier.

Coverage note — No substantial contributed material was omitted; the knowls fully capture the mathematical formulation, algorithmic procedures, optimality proofs, empirical studies, tabular benchmarks, and stated limitations of the ROCCH method and hybrid classifier.

References

1.Ali, K. M. & Pazzani, M. J. (1996). Error reduction through learning multiple descriptions. Machine Learning, 24(3), 173–202.
2.Barber, C. B., Dobkin, D. P., & Huhdanpaa, H. (1996). The quickhull algorithm for convex hulls. ACM Transactions on Mathematical Software, 22(4), 469–483. Available from ftp://geom.umn.edu/pub/software/qhull.tar.Z.
3.Beck, J. R. & Schultz, E. K. (1986). The use of ROC curves in test performance evaluation. Arch Pathol Lab Med, 110, 13–20.
4.Berry, M. J. A. & Linoff, G. (1997). Data Mining Techniques: For Marketing, Sales, and Customer Support. New York: John Wiley & Sons.
5.Blackwell, D. & Girshick, M. A. (1954). Theory of Games and Statistical Decisions. John Wiley and Sons, Inc. Republished by Dover Publications, New York, in 1979.
6.Bradley, A. P. (1997). The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30(7), 1145–1159.
7.Breiman, L. (1996). Bagging predictors. Machine Learning, 24, 123–140.
8.Breiman, L., Friedman, J., Olshen, R., & Stone, C. (1984). Classification and Regression Trees. Belmont, CA: Wadsworth International Group.
9.Catlett, J. (1995). Tailoring rulesets to misclassification costs. In Proceedings of the Fifth International Workshop on Artificial Intelligence and Statistics (pp. 88–94).
10.Cherikh, M. (1989). Optimal Decision and Detection in the Decentralized Case. Ph.D. Thesis, Case Western Reserve University.
11.Clearwater, S. & Stern, E. (1991). A rule-learning program in high energy physics event classification. Comp Physics Comm, 67, 159–182.
12.Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7), 1895–1924.
13.Domingos, P. (1999). MetaCost: A general method for making classifiers cost-sensitive. In Proceedings of the Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 155–164).
14.Domingos, P. & Pazzani, M. (1997). Beyond independence: Conditions for the optimality of the simple bayesian classifier. Machine Learning, 29, 103–130.
15.Dougherty, J., Kohavi, R., & Sahami, M. (1995). Supervised and unsupervised discretization of continuous features. In A. Prieditis & S. Russell (Eds.), Proceedings of the Twelfth International Conference on Machine Learning (pp. 194–202). San Francisco: Morgan Kaufmann.
16.Drummond, C. & Holte, R. C. (2000). Explicitly representing expected cost: An alternative to ROC representation. In R. Ramakrishnan & S. Stolfo (Eds.), Proceedings on the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
17.Egan, J. P. (1975). Signal Detection Theory and ROC Analysis. Series in Cognitition and Perception. New York: Academic Press.
18.Ezawa, K., Singh, M., & Norton, S. (1996). Learning goal oriented bayesian networks for telecommunications risk management. In L. Saitta (Ed.), Proceedings of the Thirteenth International Conference on Machine Learning (pp. 139–147). San Francisco, CA: Morgan Kaufmann.
19.Fawcett, T. & Provost, F. (1996). Combining data mining and machine learning for effective user profiling. In Simoudis, Han, & Fayyad (Eds.), Proceedings on the Second International Conference on Knowledge Discovery and Data Mining (pp. 8–13). Menlo Park, CA: AAAI Press.
20.Fawcett, T. & Provost, F. (1997). Adaptive fraud detection. Data Mining and Knowledge Discovery, 1(3), 291–316.
21.Fawcett, T. & Provost, F. (1999). Activity monitoring: Noticing interesting changes in behavior. In Chaudhuri & Madigan (Eds.), Proceedings on the Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 53–62).
22.Friedman, C. P. & Wyatt, J. C. (1997). Evaluation Methods in Medical Informatics. New York: Springer-Verlag.
23.Hanley, J. A. & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143, 29–36.
24.Klinkenberg, R. & Thorsten, J. (2000). Detecting concept drift with support vector machines. In Proceedings of the Seventeenth International Conference on Machine Learning. San Francisco: Morgan Kaufmann.
25.Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. In C. S. Mellish (Ed.), Proceedings of the 14th International Joint Conference on Artificial Intelligence (pp. 1137–1143). San Francisco: Morgan Kaufmann.
26.Kohavi, R., Sommerfield, D., & Dougherty, J. (1997). Data mining using MLC++: A machine learning library in C++. International Journal on Artificial Intelligence Tools, 6(4), 537–566. Available: http://www.sgi.com/Technology/mlc.
27.Kubat, M., Holte, R., & Matwin, S. (1998). Machine learning for the detection of oil spills in satellite radar images. Machine Learning, 30(2/3), 195–215.
28.Pazzani, M., Merz, C., Murphy, P., Ali, K., Hume, T., & Brunk, C. (1994). Reducing misclassification costs. In Proceedings of the Eleventh International Conference on Machine Learning (pp. 217–225). San Francisco: Morgan Kaufmann.
29.Provost, F. & Fawcett, T. (1997). Analysis and visualization of classifier performance: Comparison under imprecise class and cost distributions. In Proceedings of the Third International Conference on Knowledge Discovery and Data Mining (pp. 43–48). Menlo Park, CA: AAAI Press.
30.Provost, F., Fawcett, T., & Kohavi, R. (1998). The case against accuracy estimation for comparing induction algorithms. In J. Shavlik (Ed.), Proceedings of the Fifteenth International Conference on Machine Learning (pp. 445–453). San Francisco, CA: Morgan Kaufmann.
31.Quinlan, J. R. (1993). C4.5: Programs for Machine Learning. San Francisco, CA: Morgan Kaufmann.
32.Saitta, L. & Neri, F. (1998). Learning in the “Real World”. Machine Learning, 30, 133–163.
33.Salzberg, S. L. (1997). On comparing classifiers: Pitfalls to avoid and a recommended approach. Data Mining and Knowledge Discovery, 1, 317–328.
34.Srinivasan, A. (1999). Note on the location of optimal classifiers in ROC space. Technical report PRG-TR-2-99, Oxford University.
35.Stadler, W. (Ed.). (1988). Multicriteria Optimization in Engineering and in the Sciences. New York: Plenum Press.
36.Swets, J. (1988). Measuring the accuracy of diagnostic systems. Science, 240, 1285–1293.
37.Tcheng, D., Lambert, B., Lu, S. C.-Y., & Rendell, L. (1989). Building robust learning systems by computing induction and optimization. In N. S. Sridharan (Ed.), Proceedings of the Eleventh International Joint Conference on Artificial Intelligence (pp. 806–812). San Francisco, CA: Morgan Kaufmann.
38.Turney, P. (1996). Cost sensitive learning bibliography. Available: http://ai.iit.nrc.ca/bibliographies/cost-sensitive.html.
39.Weinstein, M. C. & Fineberg, H. V. (1980). Clinical Decision Analysis. Philadelphia, PA: W. B. Saunders Company.
40.Zahavi, J. & Levin, N. (1997). Issues and problems in applying neural computing to target marketing. Journal of Direct Marketing, 11(4), 63–75.

Citation

MLA

Provost, F., and T. Fawcett. “Robust Classification for Imprecise Environments”. arXiv, 2000, http://arxiv.org/abs/cs/0009007v1.

APA

Provost, F., & Fawcett, T. (2000). Robust Classification for Imprecise Environments. arXiv. http://arxiv.org/abs/cs/0009007v1

Chicago

Provost, F., and T. Fawcett. 2000. “Robust Classification for Imprecise Environments”. arXiv. http://arxiv.org/abs/cs/0009007v1.

Harvard

Provost, F. and Fawcett, T. (2000) “Robust Classification for Imprecise Environments”, arXiv [Preprint]. Available at: http://arxiv.org/abs/cs/0009007v1.

Vancouver

1. Provost F, Fawcett T (2000) Robust Classification for Imprecise Environments. arXiv

BibTeX

@article{provost2000robust,
  title = {Robust Classification for Imprecise Environments},
  author = {Provost, Foster and Fawcett, Tom},
  year = {2000},
  journal = {arXiv},
  url = {http://arxiv.org/abs/cs/0009007v1},
  eprint = {cs/0009007}
}

Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Robust Classification for Imprecise Environments

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Iso-Performance Lines in ROC Space

Knowl 2 — ROC Convex Hull Selection of Potentially Optimal Classifiers

Knowl 3 — ROCCH-Hybrid Classifier Construction and Interpolation

Knowl 4 — Optimality of ROCCH-Hybrid for Expected Cost and Accuracy

Knowl 5 — Optimality of ROCCH-Hybrid under Constrained Monotonic Metrics

Knowl 6 — Single-Pass Generation of ROC Curves from Ranked Examples

Knowl 7 — AUC Maximization of the ROCCH-Hybrid

Knowl 8 — Empirical Multi-Classifier Dominance Across Benchmark Domains

Knowl 9 — Locally Dominating Classifiers Across Iso-Performance Slope Ranges in Benchmark Domains

Knowl 10 — Limitations and Multi-Class Complexity of the ROCCH Method

References

Citation

BibTeX

Access the Paper