keyword

majority vote aggregation

Majority vote aggregation is a technique in data annotation and machine learning where multiple independent labels, judgments, or predictions for a single item are combined by selecting the option that receives the highest number of votes. Widely used in crowdsourcing and ensemble learning workflows, this method simplifies multi-annotator datasets into a single consensus label that is typically treated as the ground truth. While computationally straightforward and effective for reducing random annotator noise in objective tasks, it assumes equal competence among contributors and collapses diverse perspectives into a single outcome, which can obscure valid minority viewpoints and nuanced disagreement in subjective evaluations.

1 item

When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks

Eve Fleisig, Rediet Abebe, Dan Klein

Why you should read this

Presents a framework that predicts individual annotator judgments alongside the targeted demographic groups of text to identify when targeted populations disagree with majority-vote labels in offensive content detection.

People often disagree on subjective tasks such as determining what is offensive or toxic online, where each annotator brings their own perspective influenced by factors like culture, identity, and lived experience. In many annotation settings today, however, we ask multiple people — who may have different beliefs — to provide just one label per example, treating majority vote as ground truth. We show how training models to capture individual annotator behavior instead yields better modeling of disagreement patterns among human raters.

Source

https://aclanthology.org/2023.emnlp-main.415.pdf

Added

2026-10-02