keyword

individual rating prediction

Individual rating prediction is a machine learning approach in which a model estimates the specific score, label, or judgment that a particular human annotator would assign to a given item, rather than predicting an aggregated consensus or majority-vote outcome. In subjective tasks such as content moderation, toxicity detection, or sentiment analysis, annotator disagreement often reflects meaningful differences in demographic background, personal values, or lived experience rather than random noise. By conditioning predictions on annotator-specific attributes, identity markers, or historical labeling behaviors alongside the input content, this approach enables systems to capture nuanced disagreement patterns, model the perspectives of specific demographic groups, and preserve minority viewpoints that standard consensus aggregation might obscure.

1 item

When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks

Eve Fleisig, Rediet Abebe, Dan Klein

Why you should read this

Presents a framework that predicts individual annotator judgments alongside the targeted demographic groups of text to identify when targeted populations disagree with majority-vote labels in offensive content detection.

People often disagree on subjective tasks such as determining what is offensive or toxic online, where each annotator brings their own perspective influenced by factors like culture, identity, and lived experience. In many annotation settings today, however, we ask multiple people — who may have different beliefs — to provide just one label per example, treating majority vote as ground truth. We show how training models to capture individual annotator behavior instead yields better modeling of disagreement patterns among human raters.

Source

https://aclanthology.org/2023.emnlp-main.415.pdf

Added

2026-10-02