Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

average log-probability

Average log-probability is a metric in natural language processing and machine learning that quantifies the confidence or likelihood a model assigns to a sequence of text by calculating the arithmetic mean of the logarithmic conditional probabilities of its individual tokens. When evaluating generated text, multiplying raw token probabilities causes the total probability to shrink with sequence length, unfairly penalizing longer responses and risking numerical underflow. By converting these probabilities into logarithms, summing them across the output, and dividing by the total token count, the average log-probability creates a length-normalized score where values closer to zero indicate higher model certainty. This measurement is commonly used to score candidate answers, assess prediction calibration, and identify potentially uncertain or unreliable model generations.

1 item

Factual Confidence of LLMs: on Reliability and Robustness of Current Estimators

Factual Confidence of LLMs: on Reliability and Robustness of Current Estimators

Matéo Mahaut, Laura Aina, Paula Czarnowska, Momchil Hardalov, Thomas Müller, Lluís Màrquez

OrganizationsAmazon Web ServicesUniversitat Pompeu Fabra

Why you should read this

Evaluates five major families of factual confidence estimators across multiple large language models and tasks, revealing that internal hidden-state probes achieve superior reliability while exposing how easily model confidence fluctuates across semantically equivalent prompts.

Large Language Models (LLMs) tend to be unreliable in the factuality of their answers. To address this problem, NLP researchers have proposed a range of techniques to estimate LLM’s confidence over facts. However, due to the lack of a systematic comparison, it is not clear how the different methods compare to one another. To fill this gap, we present a survey and empirical comparison of estimators of factual confidence. We define an experimental framework allowing for fair comparison, covering both fact-verification and question answering. Our experiments across a series of LLMs indicate that trained hidden-state probes provide the most reliable confidence estimates, albeit at the expense of requiring access to weights and training data. We also conduct a deeper assessment of factual confidence by measuring the consistency of model behavior under meaning-preserving variations in the input. We find that the confidence of LLMs is often unstable across semantically equivalent inputs, suggesting that there is much room for improvement of the stability of models’ parametric knowledge. Our code is available at https://github.com/amazon-science/factual-confidence-of-llms.

Added

2026-10-03