Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

Calibration data

Calibration data refers to a distinct set of labeled examples held out from the initial model training process, used to adjust and validate the reliability of a model's predictive uncertainty and confidence estimates. Unlike training data that determines core model parameters, calibration data provides an independent benchmark to tune confidence scores, scale probabilistic outputs, or establish nonconformity thresholds. In uncertainty quantification and conformal prediction frameworks, this dataset serves to calculate empirical error distributions and determine decision boundaries, ensuring that predicted confidence levels correspond to true likelihoods and that model outputs satisfy rigorous statistical coverage guarantees.

1 item

Language Models with Conformal Factuality Guarantees

Language Models with Conformal Factuality Guarantees

Christopher Mohri, Tatsunori Hashimoto

OrganizationsAllen Institute for AIUniversity of California, San Diego

Why you should read this

Proposes a conformal prediction framework that provides rigorous statistical correctness guarantees for black-box language model outputs by progressively pruning uncertain sub-claims while preserving most generated content.

Large language models demonstrate impressive capabilities across diverse tasks yet can generate content unfaithful to external knowledge sources.To enforce controllable generation,this work introduces post-generation techniques for constrained decoding crafted specifically as defensible guards.We introduce SENSE,a framework leveraging span-level evidence to align outputs with desired alignment criteria.Rather than enforcing token-level constraints directly over entire vocabularies,SENSE operates through differentiable integration within multimodal retrieval settings,and further demonstrates how regulatory signals interact.In extensive experiments spanning three datasets,two scientific benchmarks,guided narrative completion,and factor verification pipelines grounded via Wikidata concepts.

Added

2026-10-02