keyword

ambiguity prediction

Ambiguity prediction is the computational process of identifying whether an input, such as a natural language question or user query, admits multiple valid interpretations, contexts, or answers due to underspecified information. In artificial intelligence and natural language processing systems, ambiguity prediction helps models assess uncertainty regarding user intent, allowing them to selectively abstain from answering, ask clarifying questions, or present multiple plausible responses instead of producing an inaccurate or uncalibrated single answer. Techniques for predicting ambiguity often involve evaluating internal model confidence scores, analyzing semantic feature representations, or measuring the diversity and repetition among multiple sampled outputs generated for the same input.

1 item

Selectively Answering Ambiguous Questions

Jeremy R. Cole, Michael J. Q. Zhang, Daniel Gillick, Julian Eisenschlos, Bhuwan Dhingra, Jacob Eisenstein

OrganizationsDuke UniversityGoogleUniversity of Texas at Austin

Why you should read this

Demonstrates that measuring answer consistency across repeatedly sampled outputs provides a much more reliable confidence score than model likelihoods or self-verification prompts for deciding when language models should abstain from answering ambiguous questions.

Trustworthy language models should abstain from answering questions when they do not know the answer. However, the answer to a question can be unknown for a variety of reasons. Prior research has focused on the case in which the question is clear and the answer is unambiguous but possibly unknown. But the answer to a question can also be unclear due to uncertainty of the questioner's intent or context. We investigate question answering from this perspective, focusing on answering a subset of questions with a high degree of accuracy, from a set of questions in which many are inherently ambiguous. In this setting, we find that the most reliable approach to decide when to abstain involves quantifying repetition within sampled model outputs, rather than the model's likelihood or self-verification as used in prior work. We find this to be the case across different types of uncertainty and model scales, and with or without instruction tuning. Our results suggest that sampling-based confidence scores help calibrate answers to relatively unambiguous questions, with more dramatic improvements on ambiguous questions.

Source

https://aclanthology.org/2023.emnlp-main.35.pdf

Added

2026-10-03