Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

annotation adjudication

Annotation adjudication is the process of reviewing and resolving disagreements, errors, or inconsistencies among multiple annotators to produce a reliable, high-quality standard or ground-truth dataset. In machine learning and data labeling workflows, multiple independent contributors often assign differing labels to the same data item due to ambiguity, subjective interpretation, or human error. During adjudication, a designated expert reviewer, consensus committee, or automated system evaluates conflicting annotations and their rationale to determine the correct label or to distinguish genuine human label variation from invalid errors. This reconciliation step enforces consistent guideline compliance, improves overall data quality, and establishes a trustworthy benchmark for training and evaluating computational models.

1 item

VariErr NLI: Separating Annotation Error from Human Label Variation

VariErr NLI: Separating Annotation Error from Human Label Variation

Leon Weber-Genzel, Siyao Peng, Marie-Catherine de Marneffe, Barbara Plank

OrganizationsCENTALFNRSLudwig Maximilian University of MunichMaiNLP LabMunich Center for Machine LearningUniversité catholique de Louvain

Why you should read this

Introduces the VariErr benchmark and a two-round explanation-validation methodology to disentangle genuine human label variation from true annotation errors in natural language inference, exposing critical performance gaps in automatic error detection methods.

Human label variation arises when annotators assign different labels to the same item for valid reasons, while annotation errors occur when labels are assigned for invalid reasons. These two issues are prevalent in NLP benchmarks, yet existing research has studied them in isolation. To the best of our knowledge, there exists no prior work that focuses on teasing apart error from signal, especially in cases where signal is beyond black-and-white. To fill this gap, we introduce a systematic methodology and a new dataset, VariErr (variation versus error), focusing on the NLI task in English. We propose a 2-round annotation procedure with annotators explaining each label and subsequently judging the validity of label-explanation pairs. VariErr contains 7,732 validity judgments on 1,933 explanations for 500 re-annotated MNLI items. We assess the effectiveness of various automatic error detection (AED) methods and GPTs in uncovering errors versus human label variation. We find that state-of-the-art AED methods significantly underperform GPTs and humans. While GPT-4 is the best system, it still falls short of human performance. Our methodology is applicable beyond NLI, offering fertile ground for future research on error versus plausible variation, which in turn can yield better and more trustworthy NLP systems.

Added

2026-10-04