Language Models (Mostly) Know What They Know

Saurav KadavathTom ConerlyAmanda AskellTom HenighanDawn DrainEthan PerezNicholas SchieferZac Hatfield-DoddsNova DasSarmaEli Tran-Johnson

article2022arXiv2,026 citations

Demonstrates that larger language models can reliably evaluate the correctness of their own generated statements and predict whether they know the answer to a question, providing an empirical basis for training more honest AI.

Listen

As artificial intelligence systems are deployed into high-stakes environments, ensuring their honesty and factual reliability is essential. A core prerequisite for honest AI is self-knowledge: the ability of a system to accurately evaluate whether its own reasoning is sound and to determine whether it knows the answer to a given query. The article addresses this challenge by assessing how well large language models estimate their own capabilities and evaluate the truth of their own generated responses.

The article set out to evaluate whether language models are well-calibrated when estimating the validity of their claims and to demonstrate how models can be trained to predict whether they can answer a question correctly. To investigate these questions, the authors evaluated a suite of language models ranging from 800 million to 52 billion parameters across a wide array of benchmarks, including multiple-choice datasets (such as BIG Bench and MMLU) and open-ended generative tasks spanning trivia, arithmetic, story completion, and Python code generation.

The findings establish five central conclusions. First, large models are remarkably well-calibrated on multiple-choice and True/False tasks when choices are formatted with explicit lettered options, and this calibration consistently improves with model size and few-shot prompting. Second, when models are asked to evaluate their own generated answers by predicting the probability that a response is correct, they effectively separate true statements from errors; filtering for answers with greater than 50% confidence substantially increases task accuracy. Third, self-evaluation improves markedly when the model is allowed to inspect several brainstormed sample answers before judging a specific candidate. Fourth, models can be finetuned with a specialized value head to predict the probability that they know an answer before generating it; this prediction capability generalizes to new domains, increases when relevant source text or mathematical hints are provided, and decreases when distracting hints are presented. Finally, the authors show that poorly calibrated models refined via human feedback can be restored to strong calibration using a simple probability temperature adjustment.

These results demonstrate that larger language models inherently possess significant latent self-knowledge, with verification capability improving faster than generation ability as models scale. For leaders and practitioners, this implies that organizations can deploy confidence thresholds and multi-sample self-evaluation workflows to mitigate hallucination risks and boost system reliability without retraining underlying architectures. Furthermore, the findings show that models can dynamically evaluate whether available background materials provide sufficient information to answer a prompt.

Moving forward, developers should implement few-shot True/False verification and multi-sample comparison prompts to filter generative outputs in production pipelines. However, decision-makers should note key limitations: out-of-distribution calibration remains brittle, the current approach relies on pretraining that mimics human text rather than establishing objective truth, and the techniques do not yet guard against intentional deception. Additional research and pilot testing are recommended to evaluate how these self-evaluation mechanisms function under intense reinforcement learning and across more specialized real-world domains.

Anthropic correlation analysis language model standard model Model Models True

arXiv: 2207.05221

Paper: TruthfulQA: Measuring How Models Mimic Human Falsehoods, Stephanie C. Lin et al. (2022). TruthfulQA provides the foundational evaluation methodology for measuring imitative falsehoods and model honesty that motivates the study of whether models can self-evaluate their own truthfulness.
Paper: Language Models are Few-Shot Learners, T. B. Brown et al. (2020). Demonstrates the foundational few-shot prompting paradigm and scaling behavior across question-answering tasks upon which self-evaluation and calibration methods are built.
Paper: Measuring Massive Multitask Language Understanding, Dan Hendrycks et al. (2020). Introduces the comprehensive multi-subject multiple-choice benchmark used extensively to evaluate model knowledge and calibration across scales.
Paper: Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models, BIG-bench authors (2022). Establishes broad benchmarks for scaling behavior and notes predictive calibration improvements as language models increase in parameter size.
Paper: Language Models as Knowledge Bases?, Fabio Petroni et al. (2019). Pioneered the analysis of pre-trained language models acting as factual knowledge bases, providing the conceptual basis for probing what models know.
Paper: Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?, Sewon Min et al. (2022). Analyzes how format and prompt structure drive in-context learning performance, explaining why formatting is critical for eliciting well-calibrated self-evaluations.

Paper: Metacognition in LLMs: Foundations, Progress, and Opportunities, Gabrielle Kaili-May Liu et al. (2026). Synthesizes the broader landscape of metacognition and knowledge-boundary awareness in large language models, surveying the self-evaluation and calibration paradigms investigated in the source.
Paper: Can LLMs Predict Their Own Failures? Self-Awareness via Internal Circuits, Amirhosein Ghasemabadi et al. (2025). Extends self-knowledge and failure prediction from verbalized probability prompting to probing internal network representations and attention states.
Paper: Self-Consistency Improves Chain of Thought Reasoning in Language Models, Xuezhi Wang et al. (2023). Builds directly on the insight that aggregating multiple generated candidate samples improves reasoning confidence and self-evaluation accuracy.
Paper: Self-Refine: Iterative Refinement with Self-Feedback, Aman Madaan et al. (2023). Applies the model's self-evaluation capabilities dynamically in an iterative generation, feedback, and self-correction loop.
Paper: Reflexion: language agents with verbal reinforcement learning, Noah Shinn et al. (2023). Leverages verbal self-assessment and self-reflection as episodic memory to guide autonomous agent decision-making.
Paper: Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection, Akari Asai et al. (2024). Trains models to predict their own factual knowledge gaps and dynamically retrieve external source materials via learned reflection tokens.
Paper: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Lianmin Zheng et al. (2023). Scales the concept of model self-evaluation to evaluate open-ended multi-turn dialogues and model-generated answers as a universal judge.
Paper: On-Policy Self-Distillation without Any Supervision, Yijiang Li et al.. Leverages self-generated consensus and consistency checks across independent model rollouts to achieve entirely unsupervised self-distillation.

Cover for Language Models (Mostly) Know What They Know

Abstract

We study whether language models can evaluate the validity of their own claims and predict which questions they will be able to answer correctly. We first show that larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format. Thus we can approach self-evaluation on open-ended sampling tasks by asking models to first propose answers, and then to evaluate the probability "P(True)" that their answers are correct. We find encouraging performance, calibration, and scaling for P(True) on a diverse array of tasks. Performance at self-evaluation further improves when we allow models to consider many of their own samples before predicting the validity of one specific possibility. Next, we investigate whether models can be trained to predict "P(IK)", the probability that "I know" the answer to a question, without reference to any particular proposed answer. Models perform well at predicting P(IK) and partially generalize across tasks, though they struggle with calibration of P(IK) on new tasks. The predicted P(IK) probabilities also increase appropriately in the presence of relevant source materials in the context, and in the presence of hints towards the solution of mathematical word problems. We hope these observations lay the groundwork for training more honest models, and for investigating how honesty generalizes to cases where models are trained on objectives other than the imitation of human writing.

Topics

correlation analysis language model standard model stochastic processes probability distributions

Keywords

Model Models True Comparison Examples Answer Samples Question Correct Shot Calibration

1 Introduction
1.1 Contributions
1.2 Models and Evaluation Tasks
1.3 Related Work
2 Larger Models are Calibrated on Diverse Multiple Choice Questions
3 From Calibration to Knowing What You Know
3.1 Replacing an Option with ‘None of the Above’ Harms Performance and Calibration
3.2 Models are Well-Calibrated on True/False Tasks
3.3 RLHF Policy Miscalibration Can Be Remediated with a Temperature Tuning
4 Ask the AI: Is your proposed answer True or False?
4.1 Basic Self-Evaluation
4.2 Showing Many T=1T=1 Samples Improves Self-Evaluation
5 Training Models to Predict Whether They Can Answer Questions Correctly
5.1 Evaluating P(IK) Training and Model Size Trends
5.2 Out of Distribution Generalization of P(IK)
5.3 P(IK) Generalizes to Account for Source Materials
5.4 P(IK) Generalizes to Account for Hints Towards GSM8k Solutions
5.5 Comparing Models Trained with Distinct Pretraining Distributions
6 Discussion
6.1 Limitations and Future Work
6.2 Broader Impacts
7 Contribution Statement
A Metrics, Formatting Details, and P(IK) Training
A.1 Calibration Charts
A.2 Expected Calibration Error
A.3 Brier Score
A.4 BIG Bench Default Format
A.5 Prompt for Self-Evaluation
A.6 Training for P(IK)
A.7 Formatting of Questions for P(IK)
B Discriminating What Models Know with Entropy or Loss
B.1 Loss of the Sample
B.2 Entropy of the Answer Distribution
B.3 Loss of Sampled Answers Stuffed into a Single Context
C More P(True) Evaluation Results and Details
D Mixed-Arithmetic and Function Synthesis Dataset Descriptions
References

Knowls

Knowl 1 — Value-Head Method for Estimating Language Model Self-Knowledge P(IK)
model/method
To train a language model to predict whether it can correctly answer a given question before generating an answer, an auxiliary scalar linear "value head" is attached to the final token representation of the model backbone. The target output, denoted $P(\text{IK})$ (Probability "I Know"), represents the probability that the model would produce a correct answer when sampling at temperature $T = 1$ .

For each question $Q$ formatted with a few-shot prompt, $N = 30$ independent candidate completions are sampled at $T = 1$ and evaluated against ground truth. If $k$ samples are correct and $30 - k$ are incorrect, the training set is assigned $k$ copies of $(Q, \text{IK})$ and $30 - k$ copies of $(Q, \text{IDK})$ . This represents soft probabilistic labels via collections of hard binary labels. The backbone transformer and value head are finetuned end-to-end using standard binary cross-entropy loss:

$\mathcal{L} = - \frac{1}{N} \sum_{j=1}^N \left[ y_j \log \sigma(v_j) + (1 - y_j) \log (1 - \sigma(v_j)) \right]$

where $v_j$ is the scalar value head logit computed exclusively at the final token position of prompt $Q$ , $\sigma(\cdot)$ is the sigmoid function, and $y_j \in \{0, 1\}$ is the correctness label of sample $j$ . Training utilizes a large batch size ( $7680$ ) to stabilize learning against label noise and a learning rate equal to one-third of the model pretraining learning rate.
Knowl 2 — Cross-Task Generalization and Calibration of P(IK) Classifiers
empirical result
When a 52-billion parameter autoregressive language model is finetuned with a $P(\text{IK})$ value head trained exclusively on TriviaQA, the classifier demonstrates zero-shot out-of-distribution (OOD) discrimination on diverse non-trivia tasks (evaluated by Area Under the Receiver Operating Characteristic, AUROC, for separating questions where empirical sample accuracy exceeds $0.5$ ):
- TriviaQA (In-distribution): AUROC = $0.864$ , Brier score = $0.151$
- Mixed-Arithmetic (OOD): AUROC = $0.928$ , Brier score = $0.194$
- Python Function Synthesis (OOD): AUROC = $0.687$ , Brier score = $0.164$
- GSM8k (OOD, threshold 0.5): AUROC = $0.624$ , Brier score = $0.200$
- Lambada (OOD): AUROC = $0.606$ , Brier score = $0.431$
When the 52B model is trained multi-task across TriviaQA, Mixed-Arithmetic, Python Function Synthesis, and Lambada, generalization and discrimination improve across all benchmarks (e.g., GSM8k held-out AUROC rises to $0.752$ with Brier score $0.121$ , Lambada AUROC rises to $0.853$ with Brier score $0.108$ ).

Across all evaluations, discriminative AUROC increases monotonically with parameter scale (from 800M to 52B parameters). However, out-of-distribution probability calibration is poor: models trained solely on TriviaQA exhibit severe underconfidence on OOD tasks (e.g., assigning near-zero $P(\text{IK})$ to Lambada questions that the model frequently answers correctly), whereas in-distribution predictions remain well-calibrated.
Knowl 3 — In-Context Generalization of P(IK) to Source Documents and Hints
empirical result
Although $P(\text{IK})$ classifiers are trained solely on bare question prompts without supporting context, the predicted probability $P(\text{IK})$ dynamically responds to supplementary in-context information:
1. Inclusion of Reference Text: When an obscure TriviaQA question that receives a low baseline $P(\text{IK})$ (e.g., $18\%$ for a 52B model) is prepended with a relevant Wikipedia passage, the predicted $P(\text{IK})$ rises substantially (e.g., to $78\%$ ). The average gain in $P(\text{IK})$ is inversely proportional to document length: articles under 100 tokens yield an average $P(\text{IK})$ gain of $\approx 0.29$ , whereas articles truncated at 7000 tokens yield an average gain of $\approx 0.15$ .
2. Mathematical Hints: On GSM8k word problems, prepending partial chain-of-thought solution hints increases $P(\text{IK})$ monotonically as a function of the fraction of the ground-truth solution revealed.
3. Hint Quality Sensitivity: Valid hints derived from correct solutions produce the largest $P(\text{IK})$ increases; flawed hints derived from incorrect model rollouts produce lower $P(\text{IK})$ gains; and distracting hints prepended from unrelated problems actively decrease $P(\text{IK})$ below the baseline no-hint level.

Knowl 4 — Model-Specificity of P(IK) Versus Intrinsic Question Difficulty

empirical result

To distinguish whether $P(\text{IK})$ models learn question difficulty in general or model-specific self-knowledge, two 12-billion parameter models ( $A$ and $B$ ) with identical architectures but distinct pretraining data mixtures (Model $A$ pretrained on 4 epochs of high-quality text; Model $B$ pretrained on a single epoch combined with general web crawl data) were each trained with a $P(\text{IK})$ head on their respective TriviaQA training correctness.

On the disjoint test subsets of TriviaQA where only one model generates the correct answer:

Each model assigns an approximately $6\%$ higher $P(\text{IK})$ to questions that only it answers correctly relative to questions that only the other model answers correctly, confirming that $P(\text{IK})$ captures model-specific knowledge boundaries rather than generic task hardness.

Knowl 5 — Self-Evaluation of Model Samples via P(True) and Sample Brainstorming
model/method
Language model self-evaluation on open-ended generation tasks is structured as a two-stage verification process:
1. Sample Generation: The language model first samples candidate responses $A$ for a query $Q$ at temperature $T = 1$ .
2. True/False Self-Evaluation: The question and candidate response are formatted into a multiple-choice verification template:
```
Question: <Q>
Proposed Answer: <A>
Is the proposed answer:
(A) True
(B) False
The proposed answer is:
```
The verification metric $P(\text{True})$ is defined as the normalized softmax probability assigned by the model to token (A) over token (B).
1. In-Context Sample Brainstorming: Verification accuracy and calibration improve when the model is provided with $M = 5$ candidate samples generated for question $Q$ prior to evaluating a target answer:
```
Question: <Q>
Here are some brainstormed ideas: <Sample 1>\n <Sample 2>\n <Sample 3>\n <Sample 4>\n <Sample 5>
Possible Answer: <A>
Is the possible answer:
(A) True
(B) False
The possible answer is:
```
Presenting multiple sampled alternatives gives the evaluator model in-context contrastive evidence, significantly improving AUROC and Brier scores on short-answer tasks.
Knowl 6 — Conditional Accuracy Gains and Verification Scaling with P(True)
empirical result
Filtering language model generations based on self-evaluated confidence $P(\text{True}) > 0.5$ yields large accuracy gains over unconditional sampling at temperature $T = 1$ across diverse benchmarks:
- GSM8k (52B): Base sampling accuracy $\approx 13\% \to$ conditional accuracy ( $P(\text{True}) > 0.5$ ) $\approx 38\%$ .
- Mixed-Arithmetic (52B): Base sampling accuracy $\approx 54\% \to$ conditional accuracy $\approx 77\%$ .
- Codex HumanEval (52B): Base sampling accuracy $\approx 14\% \to$ conditional accuracy $\approx 44\%$ .
- TriviaQA (52B): Base sampling accuracy $\approx 45\% \to$ conditional accuracy $\approx 67\%$ .
- Lambada (52B): Base sampling accuracy $\approx 72\% \to$ conditional accuracy $\approx 81\%$ .
The margin between conditional accuracy ( $P(\text{True}) > 0.5$ ) and unconditional base accuracy widens as parameter count scales from 800M to 52B. This divergence indicates that model verification capability scales faster with model size than raw generative ability.

While zero-shot True/False self-evaluation produces high discriminative separation (AUROC), its raw probabilities are poorly calibrated (clustering around $0.5$ ). Calibrated $P(\text{True})$ probabilities require few-shot prompting (e.g., 20-shot in-context examples).
Knowl 7 — Calibration of Autoregressive Language Models on Lettered Multiple-Choice Formats
empirical result
Pretrained autoregressive language models (ranging from 800M to 52B parameters) produce well-calibrated probabilities on multiple-choice benchmarks (such as BIG-bench, MMLU, TruthfulQA, QuALITY, and LogiQA) when prompts use explicit single-letter choice indicators:
```
Question: <Question text>
Choices:
(A) <Choice 1>
(B) <Choice 2>
(C) <Choice 3>
Answer:
```
When probabilities are computed over single token choice labels (e.g., (A), (B), (C)), predicted softmax probabilities match empirical accuracy with low Expected Calibration Error (ECE). Calibration systematically improves as:
1. Model size increases from 800M to 52B parameters.
2. Evaluation transitions from 0-shot to 5-shot in-context examples.
In contrast, prompting formats that require the model to generate the full textual string of the correct choice (such as the default BIG-bench format) exhibit substantially worse calibration due to string degeneracies and multi-token probability fragmentation.
Knowl 8 — Remediating RLHF Policy Miscalibration via Post-Hoc Temperature Scaling
empirical result
Reinforcement Learning from Human Feedback (RLHF) policies fine-tuned for helpfulness and harmlessness exhibit severe overconfidence and miscalibration at standard sampling temperature $T = 1.0$ , because reward maximization concentrates probability mass on high-reward tokens.

Applying uniform post-hoc temperature scaling to the output logit distribution:

$P_T(y_i \mid x) = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$

with a fixed global temperature $T = 2.5$ across multiple benchmarks (including MMLU standard multiple-choice, MMLU True/False reformulation, and TruthfulQA) corrects this overconfidence and restores calibration curves to align closely with the empirical identity diagonal without modifying model weights.
Knowl 9 — Calibration and Performance Degradation from 'None of the Above' Options
empirical result
Modifying multiple-choice questions by substituting the final option (e.g., option (D)) with the phrase "none of the above" severely damages both accuracy and probability calibration in pretrained language models up to 52B parameters.

On 5-shot MMLU evaluated on a 52B parameter model:
- Overall benchmark accuracy drops from $\approx 64\%$ on standard formatting to $\approx 54\%$ with "none of the above" substitutions.
- The model exhibits a strong negative bias against selecting "none of the above", assigning it systematically lower probability than its ground-truth frequency.
- Increasing in-context examples from 5-shot to 20-shot does not alleviate this bias or restore calibration.
Knowl 10 — Breakdown of Predictive Answer Entropy for Code Generation Verification
empirical result
Using the conditional Shannon entropy of model responses sampled at temperature $T = 1$ :

$H(A \mid Q) = \mathbb{E}_{A \sim P(\cdot \mid Q)} [-\log P(A \mid Q)] = \mathbb{E}_{A} \left[ \sum_{t=1}^{|A|} -\log P(a_t \mid Q, a_1, \dots, a_{t-1}) \right]$

to discriminate whether a model knows how to solve a question succeeds on short factual question answering (such as TriviaQA) but displays inverse scaling on open-ended program synthesis (Codex HumanEval).

As model parameters scale from 800M to 52B, the AUROC of $H(A \mid Q)$ for predicting solution correctness declines on Codex HumanEval. This failure occurs because capable large models synthesize diverse distinct, valid algorithmic implementations for complex programming problems. High answer entropy in code synthesis reflects expressive solution diversity rather than uncertainty or hallucination.
Knowl 11 — Metrics for Calibration and Verification: ECE, RMS Calibration Error, and Brier Score
definition
Given $N$ predictions sorted and partitioned into $M$ equal-count probability bins where bin $m$ has mean predicted probability $x_m$ and empirical correctness frequency $y_m$ , calibration and verification metrics are defined as follows:
1. Expected Calibration Error (ECE): Evaluates mean absolute error across bins (using $M = 10$ bins on top-1 predicted choices for multiple-choice tasks):
$E_{\text{ECE}} = \frac{1}{M} \sum_{m=1}^M |y_m - x_m|$
1. Root Mean Square (RMS) Calibration Error: Measures quadratic calibration error across bins:
$E_{\text{RMS}} = \sqrt{\frac{1}{M} \sum_{m=1}^M (y_m - x_m)^2}$
1. Brier Score ( $B$ ): Combines discrimination accuracy and probability calibration for binary verification predictions ( $p_i \in [0, 1]$ representing $P(\text{True})$ or $P(\text{IK})$ , and $c_i \in \{0, 1\}$ denoting true binary correctness):
$B = \frac{1}{N} \sum_{i=1}^N (p_i - c_i)^2$

Chance performance on a balanced binary task corresponds to $B = 0.25$ , with lower values indicating superior combined calibration and discrimination.

Coverage note — Omitted minor appendix details including the programmatically generated arithmetic arithmetic subset table lists and token loss calculation heuristics from stuffed sample contexts.

References

1.Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Ho, D., Hsu, J., Ibarz, J., Ichter, B., Irpan, A., Jang, E., Ruano, R. J., Jeffrey, K., Jesmonth, S., Joshi, N. J., Julian, R., Kalashnikov, D., Kuang, Y., Lee, K.-H., Levine, S., Lu, Y., Luu, L., Parada, C., Pastor, P., Quiambao, J., Rao, K., Rettinghouse, J., Reyes, D., Sermanet, P., Sievers, N., Tan, C., Toshev, A., Vanhoucke, V., Xia, F., Xiao, T., Xu, P., Xu, S., and Yan, M. (2022). Do as i can, not as i say: Grounding language in robotic affordances.
2.Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., and Kaplan, J. (2021). A general language assistant as a laboratory for alignment.
3.Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., Mann, B., and Kaplan, J. (2022). Training a helpful and harmless assistant with reinforcement learning from human feedback.
4.Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
5.Christiano, P., Cotra, A., and Xu, M. (2021). Eliciting latent knowledge.
6.Cobbe, K., Kosaraju, V., Bavarian, M., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021). Training verifiers to solve math word problems. CoRR, abs/2110.14168.
7.Desai, S. and Durrett, G. (2020). Calibration of pre-trained transformers.
8.Dinh, T., Zeng, Y., Zhang, R., Lin, Z., Gira, M., Rajput, S., Sohn, J.-y., Papailiopoulos, D., and Lee, K. (2022). Lift: Language-interfaced fine-tuning for non-language machine learning tasks.
9.Evans, O., Cotton-Barratt, O., Finnveden, L., Bales, A., Balwit, A., Wills, P., Righetti, L., and Saunders, W. (2021). Truthful ai: Developing and governing ai that does not lie.
10.Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On calibration of modern neural networks.
11.Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2021). Measuring massive multitask language understanding.
12.Hendrycks, D. and Gimpel, K. (2016). A baseline for detecting misclassified and out-of-distribution examples in neural networks.
13.Hendrycks, D., Mazeika, M., and Dietterich, T. (2018). Deep anomaly detection with outlier exposure.
14.Jiang, Z., Araki, J., Ding, H., and Neubig, G. (2021). How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering. Transactions of the Association for Computational Linguistics, 9:962–977.
15.Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. (2017). Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.
16.Lin, S., Hilton, J., and Evans, O. (2021). Truthfulqa: Measuring how models mimic human falsehoods.
17.Lin, S., Hilton, J., and Evans, O. (2022). Teaching models to express their uncertainty in words.
18.Liu, J., Cui, L., Liu, H., Huang, D., Wang, Y., and Zhang, Y. (2020). Logiqa: A challenge dataset for machine reading comprehension with logical reasoning.
19.Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., Irving, G., and McAleese, N. (2022). Teaching language models to support answers with verified quotes.
20.Mielke, S. J., Szlam, A., Boureau, Y.-L., and Dinan, E. (2020). Linguistic calibration through metacognition: aligning dialogue agent responses with expected correctness.
21.Minderer, M., Djolonga, J., Romijnders, R., Hubis, F., Zhai, X., Houlsby, N., Tran, D., and Lucic, M. (2021). Revisiting the calibration of modern neural networks. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems, volume 34, pages 15682–15694. Curran Associates, Inc.
22.Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., and Schulman, J. (2021). Webgpt: Browser-assisted question-answering with human feedback. CoRR, abs/2112.09332.
23.Nalisnick, E., Matsukawa, A., Teh, Y. W., Gorur, D., and Lakshminarayanan, B. (2019). Do deep generative models know what they don’t know? In International Conference on Learning Representations.
24.Nguyen, K. and O’Connor, B. (2015). Posterior calibration and exploratory analysis for natural language processing models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1587–1598, Lisbon, Portugal. Association for Computational Linguistics.
25.Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., Sutton, C., and Odena, A. (2021). Show your work: Scratchpads for intermediate computation with language models.
26.Ortega, P. A., Kunesch, M., DelÃ©tang, G., Genewein, T., Grau-Moya, J., Veness, J., Buchli, J., Degrave, J., Piot, B., Perolat, J., Everitt, T., Tallec, C., Parisotto, E., Erez, T., Chen, Y., Reed, S., Hutter, M., de Freitas, N., and Legg, S. (2021). Shaking the foundations: delusions in sequence models for interaction and control.
27.Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J. V., Lakshminarayanan, B., and Snoek, J. (2019). Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.
28.Pang, R. Y., Parrish, A., Joshi, N., Nangia, N., Phang, J., Chen, A., Padmakumar, V., Ma, J., Thompson, J., He, H., and Bowman, S. R. (2021). Quality: Question answering with long input texts, yes!
29.Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and FernÃ¡ndez, R. (2016). The lambada dataset: Word prediction requiring a broad discourse context.
30.Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, H. F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., van den Driessche, G., Hendricks, L. A., Rauh, M., Huang, P., Glaese, A., Welbl, J., Dathathri, S., Huang, S., Uesato, J., Mellor, J., Higgins, I., Creswell, A., McAleese, N., Wu, A., Elsen, E., Jayakumar, S. M., Buchatskaya, E., Budden, D., Sutherland, E., Simonyan, K., Paganini, M., Sifre, L., Martens, L., Li, X. L., Kuncoro, A., Nematzadeh, A., Gribovskaya, E., Donato, D., Lazaridou, A., Mensch, A., Lespiau, J., Tsimpoukelli, M., Grigorev, N., Fritz, D., Sottiaux, T., Pajarskas, M., Pohlen, T., Gong, Z., Toyama, D., de Masson d’Autume, C., Li, Y., Terzi, T., Mikulik, V., Babuschkin, I., Clark, A., de Las Casas, D., Guy, A., Jones, C., Bradbury, J., Johnson, M., Hechtman, B. A., Weidinger, L., Gabriel, I., Isaac, W. S., Lockhart, E., Osindero, S., Rimell, L., Dyer, C., Vinyals, O., Ayoub, K., Stanway, J., Bennett, L., Hassabis, D., Kavukcuoglu, K., and Irving, G. (2021). Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs/2112.11446.
31.Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
32.Varshney, N., Mishra, S., and Baral, C. (2022). Investigating selective prediction approaches across several tasks in iid, ood, and adversarial settings.
33.Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. (2022). Self-consistency improves chain of thought reasoning in language models.
34.Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. (2022). Chain of thought prompting elicits reasoning in large language models.

Citation

MLA

Kadavath, S., et al. “Language Models (Mostly) Know What They Know”. arXiv, 2022, http://arxiv.org/abs/2207.05221v4.

APA

Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., Johnston, S., El-Showk, S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., … Kaplan, J. (2022). Language Models (Mostly) Know What They Know. arXiv. http://arxiv.org/abs/2207.05221v4

Chicago

Kadavath, S., T. Conerly, A. Askell, et al. 2022. “Language Models (Mostly) Know What They Know”. arXiv. http://arxiv.org/abs/2207.05221v4.

Harvard

Kadavath, S. et al. (2022) “Language Models (Mostly) Know What They Know”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2207.05221v4.

Vancouver

1. Kadavath S, Conerly T, Askell A, et al (2022) Language Models (Mostly) Know What They Know. arXiv

BibTeX

@article{kadavath2022language,
  title = {Language Models (Mostly) Know What They Know},
  author = {Kadavath, Saurav and Conerly, Tom and Askell, Amanda and Henighan, Tom and Drain, Dawn and Perez, Ethan and Schiefer, Nicholas and Hatfield-Dodds, Zac and DasSarma, Nova and Tran-Johnson, Eli and Johnston, Scott and El-Showk, Sheer and Jones, Andy and Elhage, Nelson and Hume, Tristan and Chen, Anna and Bai, Yuntao and Bowman, Sam and Fort, Stanislav and Ganguli, Deep and Hernandez, Danny and Jacobson, Josh and Kernion, Jackson and Kravec, Shauna and Lovitt, Liane and Ndousse, Kamal and Olsson, Catherine and Ringer, Sam and Amodei, Dario and Brown, Tom and Clark, Jack and Joseph, Nicholas and Mann, Ben and McCandlish, Sam and Olah, Chris and Kaplan, Jared},
  year = {2022},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2207.05221v4},
  eprint = {2207.05221}
}

Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Evaluator Model	Questions only A gets right	Questions only B gets right
Model A Average $P(\text{IK})$	0.463	0.408
Model B Average $P(\text{IK})$	0.409	0.477

Language Models (Mostly) Know What They Know

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Value-Head Method for Estimating Language Model Self-Knowledge P(IK)

Knowl 2 — Cross-Task Generalization and Calibration of P(IK) Classifiers

Knowl 3 — In-Context Generalization of P(IK) to Source Documents and Hints

Knowl 4 — Model-Specificity of P(IK) Versus Intrinsic Question Difficulty

Knowl 5 — Self-Evaluation of Model Samples via P(True) and Sample Brainstorming

Knowl 6 — Conditional Accuracy Gains and Verification Scaling with P(True)

Knowl 7 — Calibration of Autoregressive Language Models on Lettered Multiple-Choice Formats

Knowl 8 — Remediating RLHF Policy Miscalibration via Post-Hoc Temperature Scaling

Knowl 9 — Calibration and Performance Degradation from 'None of the Above' Options

Knowl 10 — Breakdown of Predictive Answer Entropy for Code Generation Verification

Knowl 11 — Metrics for Calibration and Verification: ECE, RMS Calibration Error, and Brier Score

References

Citation

BibTeX

Access the Paper