Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

sentence-level weight assignment

Sentence-level weight assignment is a natural language processing technique that allocates numerical values or importance coefficients to individual sentences or sequence segments based on their semantic relevance, factual contribution, or structural role within a given context. Rather than treating all parts of a text uniformly, this process measures how much each sentence contributes to the overall meaning or objective of the sequence, allowing higher priority to be given to informative components over generic, redundant, or transitionary language. It is commonly applied in tasks such as text summarization, response quality evaluation, and uncertainty estimation to ensure that aggregate scores accurately reflect the core information and semantic value of generated outputs.

1 item

MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMs

MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMs

Yavuz Faruk Bakman, Duygu Nur Yaldiz, Baturalp Buyukates, Chenyang Tao, Dimitrios Dimitriadis, Salman Avestimehr

OrganizationsAmazonUniversity of Southern California

Why you should read this

Presents Meaning-Aware Response Scoring (MARS), a framework that weights each token by its semantic contribution to the answer rather than applying uniform length normalization, substantially improving uncertainty estimation and error detection across multiple large language models and question-answering benchmarks.

Generative Large Language Models (LLMs) are widely utilized for their excellence in various tasks. However, their tendency to produce inaccurate or misleading outputs poses a potential risk, particularly in high-stakes environments. Therefore, estimating the correctness of generative LLM outputs is an important task for enhanced reliability. Uncertainty Estimation (UE) in generative LLMs is an evolving domain, where SOTA probability-based methods commonly employ length-normalized scoring. In this work, we propose Meaning-Aware Response Scoring (MARS) as an alternative to length-normalized scoring for UE methods. MARS is a novel scoring function that considers the semantic contribution of each token in the generated sequence in the context of the question. We demonstrate that integrating MARS into UE methods results in a universal and significant improvement in UE performance. We conduct experiments using three distinct closed-book question-answering datasets across five popular pre-trained LLMs. Lastly, we validate the efficacy of MARS on a Medical QA dataset. Code can be found here.

Added

2026-10-04