Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

model overconfidence

Model overconfidence is a phenomenon in artificial intelligence where a model exhibits an excessively high degree of certainty in its predictions or generated answers, even when those outputs are inaccurate or incorrect. This condition arises when an algorithm fails to properly calibrate its uncertainty, leading to a disparity between its perceived or stated confidence and its true empirical accuracy. It can manifest numerically through inflated predictive probabilities or linguistically through assertive phrasing in natural language outputs. Consequently, model overconfidence poses substantial risks in real-world applications by obscuring errors, conveying a false sense of reliability, and leading users to place unwarranted trust in flawed information.

1 item

Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty

Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty

Kaitlyn Zhou, Jena D. Hwang, Xiang Ren, Maarten Sap

OrganizationsAllen Institute for AICarnegie Mellon UniversityStanford UniversityUniversity of Southern California

Why you should read this

Reveals that language models rarely express uncertainty and suffer from a 47% error rate on confident statements, tracing this miscalibration to human preference biases during post-training alignment and demonstrating the resulting risks of human overreliance.

As natural language becomes the default interface for human-AI interaction, there is a need for LMs to appropriately communicate uncertainties in downstream applications. In this work, we investigate how LMs incorporate confidence in responses via natural language and how downstream users behave in response to LM-articulated uncertainties. We examine publicly deployed models and find that LMs are reluctant to express uncertainties when answering questions even when they produce incorrect responses. LMs can be explicitly prompted to express confidences, but tend to be overconfident, resulting in high error rates (an average of 47%) among confident responses. We test the risks of LM overconfidence by conducting human experiments and show that users rely heavily on LM generations, whether or not they are marked by certainty. Lastly, we investigate the preference-annotated datasets used in post training alignment and find that humans are biased against texts with uncertainty. Our work highlights new safety harms facing human-LM interactions and proposes design recommendations and mitigating strategies moving forward.

Added

2026-10-01