Using AUC and accuracy in evaluating learning algorithms

Jin HuangC. Ling

article2005TKDE2,156 citations

In many commercial data mining operations, such as direct marketing and risk assessment, business success depends on ranking candidates by likelihood rather than making a simple yes-or-no classification. Despite this reality, predictive accuracy—the simple percentage of correct predictions—has long served as the standard benchmark for evaluating and training machine learning algorithms. However, accuracy ignores the predicted probability or confidence of an outcome, failing to reward models that rank high-probability prospects effectively. This creates a clear operational disconnect between standard evaluation metrics and real-world business objectives.

The article establishes formal criteria to evaluate performance metrics and proves that the Area Under the Receiver Operating Characteristics Curve (AUC), which measures ranking quality, is fundamentally superior to accuracy. It also re-evaluates major machine learning algorithms using AUC and quantifies how selecting models based on AUC directly impacts campaign profitability.

To conduct this evaluation, the researchers introduced formal definitions of statistical consistency (how often two metrics agree on model superiority) and discriminancy (a metric's ability to distinguish differences between models). They verified these theoretical properties across synthetic balanced, imbalanced, and multi-class datasets. They then benchmarked popular machine learning algorithms—Naive Bayes, decision trees (C4.5 and its unpruned, smoothed variant C4.4), and Support Vector Machines (SVM)—across 18 real-world UCI benchmark datasets using 10-fold cross-validation, and ran direct-marketing profit simulations.

The findings show that AUC is statistically consistent with accuracy (with consistency rates ranging from approximately 70% to 94%) and substantially more discriminating (distinguishing model quality up to 15 to 25 times more effectively). When evaluated strictly on accuracy, traditional algorithms and SVM appear virtually identical, averaging around 86% to 88% accuracy across the benchmarks. However, when evaluated using AUC, Naive Bayes (87.2%), C4.4 (86.2%), and SVM (86.0%) significantly outperform the standard decision tree algorithm C4.5 (84.5%). Furthermore, direct marketing simulations demonstrate that optimizing for AUC consistently yields higher net profits than optimizing for accuracy when targeting top customer segments (e.g., the top 15% to 25%), with the profit advantage widening as model quality increases.

These results demonstrate that long-held assumptions regarding algorithm equivalence were flawed artifacts of using accuracy as a benchmark. Because ranking quality directly drives bottom-line returns, standard accuracy introduces hidden business risk by misrepresenting model capability. Organizations relying on predictive models can achieve higher return on investment by prioritizing algorithms that excel at ranking rather than basic classification.

Decision-makers and data science teams should immediately transition from accuracy to AUC as the primary metric for model evaluation, comparison, and selection, particularly in targeted campaigns and ranking applications. Teams should also adopt Naive Bayes, C4.4, or SVM over standard C4.5 decision trees for ranking tasks and explore re-engineering training pipelines to optimize AUC directly.

While the analytical proofs and synthetic experiments provide strong confidence in these conclusions, readers should note certain limitations: the empirical comparisons discretized continuous data, utilized default kernel parameters for SVM, and simulated marketing profit under fixed cost and revenue assumptions. Further testing under highly complex, variable-cost business settings is recommended to fine-tune deployment.

business process continuous data cross validation AUC Accuracy Learning

Huang et al (2005).pdf

Paper: A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems, DAVID J. HAND et al. (2001). This paper establishes the multi-class generalization of AUC through pairwise averaging (measure M), which forms the foundational multi-class ranking metric evaluated in the source paper.
Paper: The foundations of cost-sensitive learning, Charles Elkan (2001). This work lays the theoretical foundation for cost-sensitive classification and demonstrates why threshold-independent probability ranking is necessary when misclassification costs and class distributions are uneven.
Paper: A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection, Ron Kohavi (1995). This seminal study establishes the methodological validity of stratified ten-fold cross-validation for machine learning algorithm evaluation, which the source relies on for benchmarking classifiers.
Paper: Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms, Thomas G. Dietterich (1998). This paper provides the standard statistical testing framework for comparing supervised learning algorithms and controlling type I errors across benchmark datasets.
Paper: On the Optimality of the Simple Bayesian Classifier under Zero-One Loss, Pedro M. Domingos et al. (1997). This paper provides theoretical insights into why Naive Bayes performs robustly despite inaccurate conditional probability estimates, explaining its competitive ranking and AUC performance.
Paper: SMOTE: Synthetic Minority Over-sampling Technique, N. Chawla et al. (2002). This work introduces synthetic over-sampling for imbalanced data and demonstrates its evaluation using AUC and ROC convex hulls rather than traditional classification accuracy.
Paper: In Defense of One-Vs-All Classification, Ryan Rifkin et al. (2004). This paper defends one-versus-all classification and analyzes multi-class benchmark performance of well-tuned Support Vector Machines compared to other standard models.

Paper: Predicting good probabilities with supervised learning, Alexandru Niculescu-Mizil et al. (2005). This paper extends the study of probability ranking across algorithm families by evaluating post-processing probability calibration methods like Platt scaling and isotonic regression.
Paper: An empirical comparison of supervised learning algorithms, R. Caruana et al. (2006). This study broadens the empirical benchmarking of supervised learning algorithms across nine performance metrics including AUC, squared error, and cross-entropy.
Paper: The relationship between Precision-Recall and ROC curves, Jesse Davis et al. (2006). This work deepens the mathematical understanding of ROC-based evaluation by analyzing its formal relationship and differences with Precision-Recall curves on imbalanced datasets.
Paper: Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation, David M. W. Powers (2011). This article extends performance evaluation theory by deriving chance-corrected probabilistic metrics and connecting ROC area to Informedness, Markedness, and correlation.
Paper: Do we need hundreds of classifiers to solve real world classification problems?, Manuel Fernández Delgado et al. (2014). This paper conducts an extensive real-world benchmark across hundreds of classifiers and 121 datasets to determine empirical algorithm superiority.
Paper: On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation, G. Cawley et al. (2010). This paper examines higher-level over-fitting and selection bias that arise during hyperparameter tuning and model comparison across benchmark datasets.
Paper: From RankNet to LambdaRank to LambdaMART: An Overview, Christopher J. C. Burges (2010). This overview traces the evolution of machine learning algorithms specifically engineered to optimize ranking criteria directly rather than classification accuracy.

II. C RITERIA FOR C OMPARING E VALUATION M EASURES
A. AUC vs Accuracy
B. Imbalanced Datasets
IV. C OMPARING N AIVE B AYES , D ECISION T REES , AND SVM
A. Representational Capacity
B. Comparing Naive Bayes and Decision Trees
C. Comparing Naive Bayes, Decision Trees, and SVM
D. AUC and Accuracy on Real-World Datasets
V. O PTIMIZING P ROFIT WITH AUC AND A CCURACY
VI. C ONCLUSIONS

Knowls

Knowl 1 — Statistical Criteria for Comparing Evaluation Measures
definition
To evaluate whether a performance evaluation measure $f$ is statistically better than another measure $g$ over a domain $\Psi$ of classifier outputs (e.g., ranked lists of test instances), formal pairwise comparison criteria are defined:
1. Degree of Consistency ( $C$ ): Let $R = \{(a, b) \mid a, b \in \Psi, f(a) > f(b), g(a) > g(b)\}$ be the set of pairs where both measures agree on the relative ranking of models, and $S = \{(a, b) \mid a, b \in \Psi, f(a) > f(b), g(a) < g(b)\}$ be the set of pairs where the measures contradict each other. The degree of consistency is:
$C = \frac{|R|}{|R| + |S|}, \quad 0 \le C \le 1$

$C$ is symmetric with respect to $f$ and $g$ .
1. Degree of Discriminancy ( $D$ ): Let $P = \{(a, b) \mid a, b \in \Psi, f(a) > f(b), g(a) = g(b)\}$ and $Q = \{(a, b) \mid a, b \in \Psi, g(a) > g(b), f(a) = f(b)\}$ . The degree of discriminancy of $f$ over $g$ is:
$D = \frac{|P|}{|Q|}$
1. Degree of Indifferency ( $E$ ): Let $V = \{(a, b) \mid a, b \in \Psi, a \ne b, f(a) = f(b), g(a) = g(b)\}$ and $U = \{(a, b) \mid a, b \in \Psi, a \ne b\}$ . The degree of indifferency is:
$E = \frac{|V|}{|U|}$

A measure $f$ is defined as statistically consistent and more discriminating than $g$ (intuitively, $f$ is statistically a better measure than $g$ ) if and only if $C > 0.5$ and $D > 1$ .
Knowl 2 — Theoretical Consistency and Discriminancy of AUC over Accuracy
theoretical result
Let $\Psi$ denote the domain of all possible balanced binary ranked lists of test examples (where the number of positive examples $n_0$ equals the number of negative examples $n_1$ , such that $n_0 + n_1 = n$ ). Let $\text{AUC}(a)$ and $\text{acc}(a)$ denote the Area Under the ROC Curve and classification accuracy of a ranked list $a \in \Psi$ , respectively.

Under this domain:
- Statistical Consistency: For $R = \{(a, b) \mid \text{AUC}(a) > \text{AUC}(b), \text{acc}(a) > \text{acc}(b); a, b \in \Psi\}$ and $S = \{(a, b) \mid \text{AUC}(a) < \text{AUC}(b), \text{acc}(a) > \text{acc}(b); a, b \in \Psi\}$ ,
$\frac{|R|}{|R| + |S|} > 0.5 \quad \text{or equivalently} \quad |R| > |S|$
- Statistical Discriminancy: For $P = \{(a, b) \mid \text{AUC}(a) > \text{AUC}(b), \text{acc}(a) = \text{acc}(b); a, b \in \Psi\}$ and $Q = \{(a, b) \mid \text{acc}(a) > \text{acc}(b), \text{AUC}(a) = \text{AUC}(b); a, b \in \Psi\}$ ,
$|P| > |Q| \quad \text{or equivalently} \quad D = \frac{|P|}{|Q|} > 1$

Thus, AUC is theoretically proven to be statistically consistent with and more discriminating than classification accuracy on all balanced binary ranked lists.
Knowl 3 — Empirical Consistency and Discriminancy of AUC and Accuracy across Data Distributions
empirical result
Exhaustive evaluation of all possible ranked lists on synthetic datasets confirms that AUC is statistically consistent ( $C > 0.5$ ) and more discriminating ( $D > 1$ ) than accuracy across balanced, imbalanced, and multiclass settings:
- Balanced Binary Data: For test sizes $2n \in \{4, 6, 8, 10, 12, 14, 16\}$ , degree of consistency $C$ monotonically trends toward $\approx 0.935$ , while degree of discriminancy $D$ increases from $15.5$ to $18.9$ (approaching $\approx 19$ ). The degree of indifferency $E$ drops from $0.067$ to $0.016$ .
- Imbalanced Binary Data (25% positive, 75% negative): For test set sizes $n \in \{4, 8, 12, 16\}$ with accuracy threshold fixed at the 75th percentile, $C$ ranges from $1.0$ down to $0.890$ , and $D$ ranges from $15.9$ to $21.6$ . Compared to balanced data, consistency decreases slightly while discriminancy increases.
- Varying Class Imbalance (fixed $n = 10$ ): As the positive-to-negative split varies from $5:5$ to $2:8$ (and symmetrically $8:2$ ), $C$ decreases from $0.963$ to $0.926$ while $D$ rises from $15.2$ to $22.3$ .
- Multiclass Data ( $c \in \{3, \dots, 10\}$ classes, 2 examples per class): Over 50,000 sampled lists, $C$ decreases from $0.897$ ( $c=3$ ) to $0.696$ ( $c=10$ ), while $D$ increases from $5.5$ to $25.3$ .
In all evaluated configurations, consistency remains substantially above $0.5$ and discriminancy substantially above $1.0$ .

Knowl 4 — Pairwise Verification of Consistency and Discriminancy on UCI Benchmark Datasets

empirical result

Pairwise comparisons across 180 cross-validation test folds (from 18 UCI datasets evaluated with 10-fold cross-validation) verify consistency and discriminancy on real-world data for three classifier pairs:

Metric	C4.4 vs. Naive Bayes	C4.5 vs. Naive Bayes	C4.5 vs. C4.4
$\text{acc}+ / \text{AUC}+$ or $\text{acc}- / \text{AUC}-$ ( $\|R\|$ )	84	83	45
$\text{acc}+ / \text{AUC}-$ or $\text{acc}- / \text{AUC}+$ ( $\|S\|$ )	29	31	36
Degree of Consistency ( $C$ )	0.743	0.728	0.556
$\text{acc}= / \text{AUC}\ne$ ( $\|P\|$ )	55	46	67
$\text{acc}\ne / \text{AUC}=$ ( $\|Q\|$ )	2	1	1
Degree of Discriminancy ( $D$ )	27.5	46.0	67.0
$\text{acc}= / \text{AUC}=$ ( $\|V\|$ )	10	19	31
Degree of Indifferency ( $E$ )	0.056	0.106	0.172

For every model pair, $C > 0.5$ and $D \gg 1$ . C4.5 vs. C4.4 yields the highest indifferency ( $0.172$ ) and highest discriminancy ( $67.0$ ), reflecting that both produce structurally similar trees but differ in probability calibration, which AUC detects while accuracy cannot.

Knowl 5 — Classification Accuracy and AUC Comparison of Naive Bayes, C4.5, and C4.4

empirical result

Naive Bayes, standard C4.5, and C4.4 (unpruned C4.5 with Laplace-corrected leaf probabilities) were evaluated across 18 UCI benchmark datasets using 10-fold cross-validation after entropy-based discretization of continuous attributes:

Dataset	Accuracy (%)			AUC (%)
	NB	C4.4	C4.5	NB	C4.4	C4.5
breast	$97.5 \pm 2.9$	$92.9 \pm 3.0$	$92.8 \pm 1.2$	$97.5 \pm 0.9$	$96.9 \pm 0.9$	$95.1 \pm 2.4$
cars	$86.4 \pm 3.7$	$88.9 \pm 4.0$	$85.1 \pm 3.8$	$92.8 \pm 3.3$	$94.1 \pm 3.2$	$91.4 \pm 3.5$
credit	$85.8 \pm 3.0$	$88.1 \pm 2.8$	$88.8 \pm 3.1$	$91.9 \pm 3.0$	$90.4 \pm 3.2$	$88.0 \pm 4.1$
dermatology	$98.4 \pm 1.9$	$94.0 \pm 3.5$	$94.0 \pm 4.2$	$98.6 \pm 0.1$	$97.5 \pm 1.1$	$94.6 \pm 3.3$
echocardio	$71.9 \pm 1.8$	$73.6 \pm 1.8$	$73.6 \pm 1.8$	$63.8 \pm 2.1$	$69.4 \pm 2.2$	$68.9 \pm 2.3$
ecoli	$96.7 \pm 2.2$	$96.4 \pm 3.1$	$95.5 \pm 3.9$	$97.0 \pm 1.1$	$97.0 \pm 1.0$	$94.3 \pm 3.6$
glass	$71.8 \pm 2.4$	$73.3 \pm 3.9$	$73.3 \pm 3.0$	$76.1 \pm 2.4$	$73.1 \pm 2.6$	$71.3 \pm 3.3$
heart	$80.8 \pm 7.3$	$78.9 \pm 7.6$	$81.2 \pm 5.6$	$82.7 \pm 6.1$	$80.1 \pm 7.8$	$76.2 \pm 7.0$
hepatitis	$83.0 \pm 6.2$	$81.3 \pm 4.4$	$84.0 \pm 4.0$	$76.5 \pm 4.4$	$62.9 \pm 8.2$	$59.2 \pm 6.8$
import	$96.1 \pm 3.9$	$100.0 \pm 0.0$	$100.0 \pm 0.0$	$91.7 \pm 4.5$	$94.4 \pm 2.0$	$95.1 \pm 2.6$
iris	$95.3 \pm 4.5$	$95.3 \pm 4.5$	$95.3 \pm 4.5$	$94.2 \pm 3.4$	$91.8 \pm 3.8$	$92.4 \pm 4.6$
liver	$62.3 \pm 5.7$	$60.5 \pm 4.8$	$61.1 \pm 4.9$	$61.5 \pm 5.9$	$59.6 \pm 5.7$	$60.5 \pm 5.0$
mushroom	$97.2 \pm 0.8$	$100.0 \pm 0.0$	$100.0 \pm 0.0$	$99.7 \pm 0.1$	$99.9 \pm 0.0$	$99.9 \pm 0.0$
pima	$71.4 \pm 5.8$	$71.9 \pm 7.1$	$71.7 \pm 6.8$	$75.9 \pm 4.2$	$73.4 \pm 7.3$	$72.4 \pm 7.4$
solar	$74.0 \pm 3.2$	$73.0 \pm 3.1$	$73.9 \pm 2.1$	$88.7 \pm 1.7$	$87.7 \pm 1.9$	$85.2 \pm 2.8$
thyroid	$95.7 \pm 1.1$	$96.0 \pm 1.1$	$96.6 \pm 1.1$	$94.9 \pm 1.8$	$94.3 \pm 2.6$	$92.1 \pm 5.5$
voting	$91.4 \pm 5.6$	$95.7 \pm 4.6$	$96.6 \pm 3.9$	$91.4 \pm 3.7$	$95.2 \pm 2.2$	$93.4 \pm 3.7$
wine	$98.9 \pm 2.4$	$95.0 \pm 4.9$	$95.5 \pm 5.1$	$95.3 \pm 1.8$	$94.4 \pm 1.2$	$91.6 \pm 4.0$
Average	86.4	86.4	86.6	87.2	86.2	84.5

Two-tailed paired t-tests ( $p < 0.05$ ) establish:

In predictive accuracy, differences between NB ( $86.4\%$ ) and C4.4 ( $86.4\%$ ), NB and C4.5 ( $86.6\%$ ), and C4.4 and C4.5 are not statistically significant.
In AUC, both Naive Bayes ( $87.2\%$ ) and C4.4 ( $86.2\%$ ) statistically significantly outperform C4.5 ( $84.5\%$ ). The difference between Naive Bayes and C4.4 is not statistically significant.

Knowl 6 — Benchmark Comparison of Support Vector Machines against Naive Bayes and Decision Trees

empirical result

Support Vector Machines (SVM with Gaussian kernel, tuned via 3-fold cross-validation grid search on training folds over $C \in [2^{-5}, 2^{15}]$ and $\gamma \in [2^{-15}, 2^3]$ using LIBSVM) were evaluated on 13 binary UCI datasets with entropy-discretized attributes:

Dataset	Accuracy (%)	AUC (%)
breast	$96.5 \pm 2.3$	$97.3 \pm 1.3$
cars	$97.0 \pm 1.3$	$98.6 \pm 0.4$
credit	$86.4 \pm 2.9$	$90.4 \pm 3.0$
echocardio	$73.6 \pm 1.8$	$71.5 \pm 2.0$
ecoli	$96.4 \pm 3.1$	$95.0 \pm 2.8$
heart	$79.7 \pm 8.2$	$82.1 \pm 8.3$
hepatitis	$85.8 \pm 4.2$	$64.2 \pm 8.7$
import	$100.0 \pm 0.0$	$93.8 \pm 0.6$
liver	$60.5 \pm 4.8$	$61.6 \pm 5.6$
mushroom	$99.9 \pm 0.1$	$99.9 \pm 0.0$
pima	$72.2 \pm 6.3$	$72.2 \pm 7.5$
thyroid	$96.7 \pm 1.3$	$95.8 \pm 3.3$
voting	$97.0 \pm 3.5$	$95.3 \pm 0.7$
SVM Average	87.8	86.0
NB Average (13 datasets)	85.9	86.0
C4.4 Average (13 datasets)	86.5	85.2
C4.5 Average (13 datasets)	86.7	83.6

Paired t-tests ( $p < 0.05$ ) demonstrate:

In predictive accuracy, the higher mean of SVM ( $87.8\%$ ) over NB ( $85.9\%$ ), C4.4 ( $86.5\%$ ), and C4.5 ( $86.7\%$ ) is not statistically significant.
In AUC, SVM ( $86.0\%$ ), Naive Bayes ( $86.0\%$ ), and C4.4 ( $85.2\%$ ) show no statistically significant difference from one another, and all three achieve statistically significantly higher AUC than C4.5 ( $83.6\%$ ).

Knowl 7 — Representational Ranking Capacity of Decision Trees versus Naive Bayes
theoretical result
Decision trees and Naive Bayes exhibit different fundamental limits in their capacity to produce accurate instance rankings:
- Decision Trees: The posterior probability assigned to an instance is determined solely by the class distribution of the terminal leaf node into which it falls. Consequently, all instances mapped to the same leaf receive identical probabilities and are ranked arbitrarily. Pruning algorithms (such as in standard C4.5) reduce tree size to prevent classification error overfitting, which reduces the number of distinct leaves and worsens probability ties. Conversely, larger unpruned trees have sparse leaves that produce uncalibrated probabilities ( $0\%$ or $100\%$ ), requiring probability smoothing (e.g., Laplace correction in C4.4) to restore reliable ranking.
- Naive Bayes: For an instance described by $n$ attributes $(a_1, \dots, a_n)$ and class $c$ , posterior probability estimation $P(c \mid a_1, \dots, a_n) \propto P(c) \prod_{i=1}^n P(a_i \mid c)$ uses only $2n + 1$ parameters but can generate up to $2^n$ distinct posterior probability values. This multiplicative composition enables Naive Bayes to produce fine-grained probability rankings without the leaf-discretization bottleneck inherent to decision trees.
Knowl 8 — Direct Marketing Profit Optimization via AUC versus Accuracy
empirical result
In a direct marketing simulation where only the top $X\%$ ( $X < 50$ ) of prospects are targeted with fixed promotion revenue and cost per contact on balanced customer datasets ( $n=20$ examples evaluated over 1,000,000 random ranking permutations):
- For any evaluation score above $50\%$ , the expected net profit produced by a classifier with a given AUC is strictly greater than the expected net profit produced by a classifier with an equal accuracy value.
- The profit advantage of AUC over accuracy widens as metric values increase (e.g., at a $15\%$ cut-off, the profit difference between AUC and accuracy is $2.7$ at $60\%$ , $5.0$ at $70\%$ , $7.0$ at $80\%$ , and $4.8$ at $90\%$ ).
- Smaller targeted promotion fractions (e.g., $15\%$ vs. $25\%$ ) yield larger profit advantages for AUC over accuracy.
- The slope of the profit curve with respect to AUC is steeper than that of accuracy, showing that a unit gain in AUC produces greater incremental profit than an equal unit gain in accuracy.
- Maximum campaign profit at a $15\%$ cut-off is achieved when AUC reaches $\ge 93\%$ , whereas achieving maximum profit with accuracy requires $100\%$ accuracy.
Knowl 9 — Formulation of Binary and Multiclass AUC from Probability Rankings
equation
For a binary classification problem on a dataset with $n_0$ positive examples and $n_1$ negative examples, let $r_i$ be the rank of the $i$ -th positive example in the list sorted in ascending order of predicted positive probability, and let $S_0 = \sum_{i=1}^{n_0} r_i$ . The AUC estimate $\hat{A}$ is given by:

$\hat{A} = \frac{S_0 - n_0(n_0 + 1)/2}{n_0 n_1}$

For a multiclass classification problem with $c$ classes, where each example is assigned predicted class probabilities $(p_1, p_2, \dots, p_c)$ , multiclass AUC is computed by averaging pairwise AUCs (Hand and Till generalization). For each pair of classes $i < j$ :
1. Subsetting only instances belonging to classes $i$ and $j$ , rank them by $p_i$ to calculate $\text{AUC}(i, j)$ , and rank them by $p_j$ to calculate $\text{AUC}(j, i)$ .
2. Compute the symmetric pairwise AUC:
$\hat{\text{AUC}}(i, j) = \frac{\text{AUC}(i, j) + \text{AUC}(j, i)}{2}$
1. Average across all $\binom{c}{2}$ pairs:
$\text{AUC}_{\text{multi}} = \frac{2}{c(c - 1)} \sum_{i < j} \hat{\text{AUC}}(i, j)$
Knowl 10 — Formulation of C4.4 Probability Estimation via Unpruned Trees with Laplace Correction
model/method
The C4.4 algorithm modifies C4.5 to improve AUC and probability ranking through two adjustments:
1. Unpruned Decision Tree Construction: Tree pruning is disabled, leaving the decision tree fully expanded. This maximizes the number of leaf nodes, minimizing the number of instances mapped to identical leaves and maximizing the number of distinct probability ranks.
2. Laplace Correction for Leaf Probabilities: To mitigate extreme and unreliable probability estimates caused by small leaf sample sizes in unpruned trees, leaf probability estimates are smoothed using Laplace correction:
$P(c = k \mid \text{leaf}) = \frac{n_k + 1}{N + C}$

where $n_k$ is the number of training instances of class $k$ at that leaf node, $N$ is the total number of training instances at the leaf, and $C$ is the number of target classes.

Coverage note — No substantial contributed material was omitted. All theoretical criteria, empirical verifications, model comparisons (Naive Bayes, C4.5, C4.4, SVM), and profit optimization analyses were fully represented.

References

1.C. Blake and C. Merz. UCI repository of machine learning databases. http://www.ics.uci.edu/~mlearn/MLRepository.html, 1998. University of California, Irvine, Dept. of Information and Computer Sciences.
2.B. Boser, I. Guyon, and V. Vapnik. A training algorithm for optimal margin classifiers. In Proceedings of the Fifth Conference on Computational Learning Theory , pages 144–152, 1992.
3.A. P. Bradley. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30:1145–1159, 1997.
4.M. Brown, W. Grundy, D. Lin, and N. C. et al. Knowledge-based analysis of microarray gene expression data using support vector machines. In Proceedings of the National Academy of Sciences, pages 262–267, 2000.
5.C. J. Burges. A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2(2):121–167, 1998.
6.C. C. Chang and C. Lin. Libsvm: A library for support vector machines (version 2.4), 2003.
7.W. W. Cohen, R. E. Schapire, and Y. Singer. Learning to order things. Journal of Artificial Intelligence Research, 10:243–270, 1999.
8.N. Cristianini and J. Shawe-Taylor. An Introduction to Support Vector Machines. Cambridge University Press, 2000.
9.P. Domingos and M. Pazzani. Beyond independence: conditions for the optimality of the simple Bayesian classifier. In Proceedings of the Thirteenth International Conference on Machine Learning , pages 105 – 112, 1996.
10.U. Fayyad and K. Irani. Multi-interval discretization of continuous-valued attributes for classification learning. In Proceedings of Thirteenth International Joint Conference on Artificial Intelligence , pages 1022–1027. Morgan Kaufmann, 1993.
11.C. Ferri, P. A. Flach, and J. Hernandez-Orallo. Learning decision trees using the area under the ROC curve. In Proceedings of the Nineteenth International Conference on Machine Learning (ICML 2002), pages 139–146, 2002.
12.D. J. Hand and R. J. Till. A simple generalisation of the area under the ROC curve for multiple class classification problems. Machine Learning, 45:171–186, 2001.
13.T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Springer, 2001.
14.C. Hsu and C. Lin. A comparison on methods for multi-class support vector machines. Technical report, Department of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan, 2001.
15.J. Huang, J. Lu, and C. X. Ling. Comparing naive bayes, decision trees, and svm using accuracy and auc. In Proceedings of the 3rd International Conference on Data Mining(ICDM-2003), page To appear, 2003.
16.T. Joachims. Text categorization with support vector machines: Learning with many relevant features. In Proceedings of European Conference on Machine Learning , pages 137–142, 1998.
17.I. Kononenko. Comparison of inductive and naive Bayesian learning approaches to automatic knowledge acquisition. In B. Wielinga, editor, Current Trends in Knowledge Acquisition. IOS Press, 1990.
18.P. Langley, W. Iba, and K. Thomas. An analysis of Bayesian classifiers. In Proceedings of the Tenth National Conference of Artificial Intelligence, pages 223–228. AAAI Press, 1992.
19.Y. Lin. Support vector machines and the bayes rule in classification. Data Mining and Knowledge Discovery, 6(3):259–275, 2002.
20.C. Ling and C. Li. Data mining for direct marketing - specific problems and solutions. In Proceedings of Fourth International Conference on Knowledge Discovery and Data Mining (KDD-98), pages 73–79, 1998.
21.C. Ling and J. Yan. Decision tree with better ranking. In Proceedings of 2003 International Conference on Machine Learning (ICML’2003), 2003.
22.C. X. Ling, J. Huang, and H. Zhang. AUC: a statistically consistent and more discriminating measure than accuracy. In Proceedings of 18th International Conference on Artificial Intelligence (IJCAI-2003), pages 329–341, 2003.
23.C. X. Ling and H. Zhang. Toward Bayesian classifiers with accurate probabilities. In Proceedings of the Sixth Pacific-Asia Conference on KDD, pages 123–134. Springer, 2002.
24.H. Liu, F. Hussain, C. L. Tan, and M. Dash. Discretization: An enabling technique. Data Mining and Knowledge Discovery, 6(4):393–423, 2002.
25.D. Meyer, F. Leisch, and K. Hornik. Benchmarking support vector machines. Technical report, Vienna University of Economics and Business Administration, 2002.
26.F. Provost and P. Domingos. Tree induction for probability-based ranking. Machine Learning, 2003. To appear.
27.F. Provost and T. Fawcett. Analysis and visualization of classifier performance: comparison under imprecise class and cost distribution. In Proceedings of the Third International Conference on Knowledge Discovery and Data Mining , pages 43–48. AAAI Press, 1997.
28.F. Provost, T. Fawcett, and R. Kohavi. The case against accuracy estimation for comparing induction algorithms. In Proceedings of the Fifteenth International Conference on Machine Learning , pages 445–453. Morgan Kaufmann, 1998.
29.J. Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann: San Mateo, CA, 1993.
30.B. Scholkopf and A. Smola. Learning with Kernels. MIT Press, 2002.
31.P. Smyth, A. Gray, and U. Fayyad. Retrofitting decision tree classifiers using kernel density estimation. In Proceedings of the 12th International Conference on machine Learning , pages 506–514, 1995.
32.J. A. K. Suykens and J. Vandewalle. Multiclass least squares support vector machines. In IJCNN’99 International Joint Conference on Neural Networks, Washington, DC, 1999.
33.V. Vapnik. Statistical Learning Theory. Wiley, NY, 1998.

Citation

MLA

Jin Huang, and C. X. Ling. “Using AUC and Accuracy in Evaluating Learning Algorithms”. IEEE Transactions on Knowledge and Data Engineering, vol. 17, no. 3, 2005, pp. 299–310, https://doi.org/10.1109/TKDE.2005.50.

APA

Jin Huang, & Ling, C. X. (2005). Using AUC and accuracy in evaluating learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 17(3), 299–310. https://doi.org/10.1109/TKDE.2005.50

Chicago

Jin Huang, and C. X. Ling. 2005. “Using AUC and Accuracy in Evaluating Learning Algorithms”. IEEE Transactions on Knowledge and Data Engineering 17 (3): 299–310. https://doi.org/10.1109/TKDE.2005.50.

Harvard

Jin Huang and Ling, C.X. (2005) “Using AUC and accuracy in evaluating learning algorithms”, IEEE Transactions on Knowledge and Data Engineering, 17(3), pp. 299–310. Available at: https://doi.org/10.1109/TKDE.2005.50.

Vancouver

1. Jin Huang, Ling CX (2005) Using AUC and accuracy in evaluating learning algorithms. IEEE Transactions on Knowledge and Data Engineering 17:299–310

BibTeX

@article{Jin_Huang_2005, title={Using AUC and accuracy in evaluating learning algorithms}, volume={17}, ISSN={1041-4347}, url={http://dx.doi.org/10.1109/TKDE.2005.50}, DOI={10.1109/tkde.2005.50}, number={3}, journal={IEEE Transactions on Knowledge and Data Engineering}, publisher={Institute of Electrical and Electronics Engineers (IEEE)}, author={Jin Huang and Ling, C.X.}, year={2005}, month=Mar, pages={299–310} }

Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Using AUC and accuracy in evaluating learning algorithms

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Statistical Criteria for Comparing Evaluation Measures

Knowl 2 — Theoretical Consistency and Discriminancy of AUC over Accuracy

Knowl 3 — Empirical Consistency and Discriminancy of AUC and Accuracy across Data Distributions

Knowl 4 — Pairwise Verification of Consistency and Discriminancy on UCI Benchmark Datasets

Knowl 5 — Classification Accuracy and AUC Comparison of Naive Bayes, C4.5, and C4.4

Knowl 6 — Benchmark Comparison of Support Vector Machines against Naive Bayes and Decision Trees

Knowl 7 — Representational Ranking Capacity of Decision Trees versus Naive Bayes

Knowl 8 — Direct Marketing Profit Optimization via AUC versus Accuracy

Knowl 9 — Formulation of Binary and Multiclass AUC from Probability Rankings

Knowl 10 — Formulation of C4.4 Probability Estimation via Unpruned Trees with Laplace Correction

References

Citation

BibTeX

Access the Paper