Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

annotator uncertainty

Annotator uncertainty refers to the lack of confidence, hesitation, or ambiguity experienced by a human labeler when assigning a label, category, or interpretation to a given piece of data. This phenomenon typically occurs when the data instance is inherently ambiguous, context is incomplete, multiple interpretations are plausible, or the annotation guidelines leave room for subjective judgment. Unlike careless mistakes or random noise, annotator uncertainty is a legitimate cognitive response to difficult or subjective tasks that contributes significantly to human label variation. In machine learning and data curation, understanding annotator uncertainty allows practitioners to account for boundary cases, evaluate label reliability, and model distributions of possible labels rather than forcing an assumption of a single unequivocal ground truth.

1 item

The "Problem" of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation

The "Problem" of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation

Barbara Plank

OrganizationsLMU MunichLudwig Maximilian University of MunichMaiNLP LabMunich Center for Machine Learning

Why you should read this

Argues that human annotation variation represents meaningful subjectivity rather than noise, synthesizing its impact across data collection, modeling, and evaluation while compiling a unified repository of un-aggregated datasets to guide future machine learning research.

Human variation in labeling is often considered noise. Annotation projects for machine learning (ML) aim at minimizing human label variation, with the assumption to maximize data quality and in turn optimize and maximize machine learning metrics. However, this conventional practice assumes that there exists a ground truth, and neglects that there exists genuine human variation in labeling due to disagreement, subjectivity in annotation or multiple plausible answers. In this position paper, we argue that this big open problem of human label variation persists and critically needs more attention to move our field forward. This is because human label variation impacts all stages of the ML pipeline: data, modeling and evaluation. However, few works consider all of these dimensions jointly; and existing research is fragmented. We reconcile different previously proposed notions of human label variation, provide a repository of publicly-available datasets with un-aggregated labels, depict approaches proposed so far, identify gaps and suggest ways forward. As datasets are becoming increasingly available, we hope that this synthesized view on the “problem” will lead to an open discussion on possible strategies to devise fundamentally new directions.

Added

2026-10-01