Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

accuracy rejection curves

An accuracy rejection curve is a graphical evaluation tool used in machine learning to show how a classification model accuracy improves as varying proportions of its most uncertain predictions are withheld from evaluation. To generate the curve, model predictions are ranked according to estimated uncertainty or confidence scores, and accuracy is repeatedly measured on the remaining retained instances across increasing rejection rates spanning from zero to one hundred percent. This representation illustrates the operational trade-off between data coverage and predictive correctness, providing an effective way to assess uncertainty quantification techniques, identify potential misclassifications, and select threshold settings where automated systems can safely abstain and defer ambiguous decisions to secondary review.

1 item

Uncertainty Estimation of Transformer Predictions for Misclassification Detection

Uncertainty Estimation of Transformer Predictions for Misclassification Detection

Artem Vazhentsev, Gleb Kuzmin, Artem Shelmanov, Akim Tsvigun, Evgenii Tsymbalov, Kirill Fedyanin, Maxim Panov, Alexander Panchenko, Gleb Gusev, Mikhail Burtsev, Manvel Avetisian, Leonid Zhukov

Why you should read this

Develops computationally efficient uncertainty estimation methods for Transformer models in text classification and named entity recognition, demonstrating that a spectral-normalized Mahalanobis distance approach can rival or exceed heavy deep ensembles at detecting misclassifications.

Uncertainty estimation (UE) of model predictions is a crucial step for a variety of tasks such as active learning, misclassification detection, adversarial attack detection, out-of-distribution detection, etc. Most of the works on modeling the uncertainty of deep neural networks evaluate these methods on image classification tasks. Little attention has been paid to UE in natural language processing. To fill this gap, we perform a vast empirical investigation of state-of-the-art UE methods for Transformer models on misclassification detection in named entity recognition and text classification tasks and propose two computationally efficient modifications, one of which approaches or even outperforms computationally intensive methods1.

Added

2026-10-03