Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

human disagreement

Human disagreement refers to the divergence in judgments, interpretations, or labels provided by multiple human annotators when evaluating the same instance of data. In machine learning and data science, this variation frequently arises from task ambiguity, linguistic complexity, or subjective differences in perspective rather than simple annotator error. While traditional evaluation frameworks historically treated these discrepancies as noise to be resolved through majority voting or consensus filtering, human disagreement is increasingly recognized as a meaningful signal that captures genuine uncertainty, motivating approaches that evaluate models against the full distribution of human responses rather than enforcing a single artificial ground truth.

1 item

Stop Measuring Calibration When Humans Disagree

Stop Measuring Calibration When Humans Disagree

Joris Baan, Wilker Aziz, Barbara Plank, Raquel Fernández

OrganizationsIT University of CopenhagenLudwig Maximilian University of MunichMunich Center for Machine LearningUniversity of Amsterdam

Why you should read this

Demonstrates the theoretical flaws of standard calibration metrics when evaluated against majority labels on ambiguous tasks and proposes instance-level measures that directly align model confidence with full distributions of human judgment.

Calibration is a popular framework to evaluate whether a classifier knows when it does not know—i.e., its predictive probabilities are a good indication of how likely a prediction is to be correct. Correctness is commonly estimated against the human majority class. Recently, calibration to human majority has been measured on tasks where humans inherently disagree about which class applies. We show that measuring calibration to human majority given inherent disagreements is theoretically problematic, demonstrate this empirically on the ChaosNLI dataset, and derive several instance-level measures of calibration that capture key statistical properties of human judgements—class frequency, ranking and entropy.1

Added

2026-10-02