Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

consistency-based estimation

Consistency-based estimation is a method for measuring the confidence or uncertainty of an artificial intelligence model by evaluating how consistently it produces the same or semantically equivalent output across multiple independent generations. In natural language processing and large language models, this technique operates primarily in a black-box setting where the model is queried repeatedly for the same task using stochastic sampling, perturbed prompts, or varied reasoning paths. The degree of consensus, frequency of the majority answer, or semantic similarity among the resulting candidate outputs serves as a proxy for the likelihood that the response is correct. Because it relies solely on the statistical agreement of observable generated text rather than internal parameters, hidden activations, or raw token probabilities, consistency-based estimation is widely applied to calibrate model reliability and quantify uncertainty across both open and closed-access systems.

1 item

A Survey of Confidence Estimation and Calibration in Large Language Models

A Survey of Confidence Estimation and Calibration in Large Language Models

Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, Iryna Gurevych

OrganizationsMohamed bin Zayed University of Artificial IntelligenceTechnische Universität Darmstadt

Why you should read this

Presents a structured taxonomy of methods, metrics, and applications for estimating and calibrating large language model confidence to mitigate factual errors and improve generation reliability.

Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks in various domains. Despite their impressive performance, they can be unreliable due to factual errors in their generations. Assessing their confidence and calibrating them across different tasks can help mitigate risks and enable LLMs to produce better generations. There has been a lot of recent research aiming to address this, but there has been no comprehensive overview to organize it and to outline the main lessons learned. The present survey aims to bridge this gap. In particular, we outline the challenges and we summarize recent technical advancements for LLM confidence estimation and calibration. We further discuss their applications and suggest promising directions for future work.

Added

2026-09-26