Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

evaluation bias

Evaluation bias refers to systematic skew or prejudice in the assessment, scoring, or ranking of outputs generated by machine learning systems, leading to distorted conclusions about their relative performance. This phenomenon occurs when an evaluator, whether an automated model acting as a judge or a human assessor, is influenced by extraneous factors rather than the intrinsic quality and accuracy of the evaluated content. Common manifestations include positional bias, where preferences are shaped by the presentation order of candidate responses, verbosity bias that disproportionately favors longer answers, and self-preference, where evaluators systematically rate outputs generated by their own architecture higher. These unintended distortions undermine the reliability of benchmark results, resulting in misaligned comparisons and flawed assessments of model capabilities.

1 item

Large Language Models are not Fair Evaluators

Large Language Models are not Fair Evaluators

Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, Zhifang Sui

OrganizationsPeking UniversityTencentUniversity of Hong Kong

Why you should read this

Reveals that using large language models as judges introduces severe positional bias that distorts model rankings, and provides effective calibration strategies to align automated evaluations with human judgments.

In this paper, we uncover a positional bias in the evaluation paradigm of adopting large language models (LLMs), e.g., GPT-4, as a referee to score and compare the quality of responses generated by candidate models. We find that the quality ranking of candidate responses can be easily hacked by simply altering their order of appearance in the context. This manipulation allows us to skew the evaluation result, making one model appear considerably superior to the other, e.g., Vicuna-13B could beat ChatGPT on 66 over 80 tested queries with ChatGPT as an evaluator. We propose a simple yet effective calibration framework to address our discovered positional bias. To evaluate the effectiveness of our framework, we manually annotate the “win/tie/lose” outcomes of responses from ChatGPT and Vicuna-13B in the Vicuna Benchmark’s question prompt. Extensive experiments demonstrate that our approach successfully alleviates evaluation bias, resulting in closer alignment with human judgments. To facilitate future research on more robust large language model comparison, we integrate the techniques in the paper into an easy-to-use toolkit FairEval, along with the human annotations 1.

Added

2026-09-28