Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

informative disagreement

Informative disagreement refers to variation in judgment or labeling among human evaluators that reflects genuine differences in perspective, subjective interpretation, or inherent ambiguity in the data, rather than random error or noise. In data annotation and computational evaluation, conventional practices frequently treat discrepancies between annotators as flaws to be eliminated or collapsed into a single consensus label. By contrast, informative disagreement treats these divergent assessments as valuable signals that capture legitimate alternative meanings, varied cultural or demographic viewpoints, and task complexity. Accounting for such disagreement enables predictive systems and evaluation pipelines to represent real-world uncertainty, preserve diverse viewpoints, and handle nuanced or subjective concepts more faithfully.

1 item

The "Problem" of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation

The "Problem" of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation

Barbara Plank

OrganizationsLMU MunichLudwig Maximilian University of MunichMaiNLP LabMunich Center for Machine Learning

Why you should read this

Argues that human annotation variation represents meaningful subjectivity rather than noise, synthesizing its impact across data collection, modeling, and evaluation while compiling a unified repository of un-aggregated datasets to guide future machine learning research.

Human variation in labeling is often considered noise. Annotation projects for machine learning (ML) aim at minimizing human label variation, with the assumption to maximize data quality and in turn optimize and maximize machine learning metrics. However, this conventional practice assumes that there exists a ground truth, and neglects that there exists genuine human variation in labeling due to disagreement, subjectivity in annotation or multiple plausible answers. In this position paper, we argue that this big open problem of human label variation persists and critically needs more attention to move our field forward. This is because human label variation impacts all stages of the ML pipeline: data, modeling and evaluation. However, few works consider all of these dimensions jointly; and existing research is fragmented. We reconcile different previously proposed notions of human label variation, provide a repository of publicly-available datasets with un-aggregated labels, depict approaches proposed so far, identify gaps and suggest ways forward. As datasets are becoming increasingly available, we hope that this synthesized view on the “problem” will lead to an open discussion on possible strategies to devise fundamentally new directions.

Added

2026-10-01