CROWDSOURCING A WORD–EMOTION ASSOCIATION LEXICON
Saif M. MohammadPeter D. Turney
Presents a practical crowdsourcing methodology for constructing large-scale word-emotion lexicons, proving that sense-verification questions and association-based framing substantially increase annotation quality and inter-annotator agreement.
The article addresses the scarcity of large, high-quality emotion lexicons compared to polarity resources, which limits progress in sentiment analysis for applications such as customer relations management, search engines, tutoring systems, and literary analysis. Emotions manifest in language through words, yet manual expert annotation has proven too costly and slow to build comprehensive resources.
The work set out to create a large English term–emotion association lexicon quickly and inexpensively by harnessing crowdsourcing while solving quality-control challenges inherent to the method.
Researchers used Amazon’s Mechanical Turk to obtain annotations for more than 10,000 word–sense pairs drawn from frequent unigrams, bigrams, the General Inquirer, and the WordNet Affect Lexicon. They presented each term with a word-choice question to convey sense and filter out unfamiliar or malicious responses, then collected five independent ratings per term for eight basic emotions and for positive or negative polarity. Pilot experiments compared question phrasing, and post-processing retained only high-quality assignments.
The resulting lexicon, EmoLex, shows that roughly 36 percent of terms evoke at least one emotion and 30 percent carry strong positive or negative polarity. Adjectives and adverbs are most often emotive; trust and joy appear most frequently among the eight emotions. Annotator agreement reached fair to substantial levels, with at least four of five workers concurring on the majority of terms. Annotations aligned closely with existing gold-standard resources, and about 9 percent of terms were judged to name emotions directly. Certain emotions such as anger and sadness co-occur more often than others.
These results demonstrate that carefully designed crowdsourcing can produce reliable emotion data at low cost, enabling practical systems that detect customer anger, identify trusted products, or track emotional arcs in text. The findings also clarify which parts of speech and which emotion pairs warrant priority in downstream models.
The authors recommend expanding coverage to 40,000 terms, building parallel lexicons in other languages, testing the resource in live applications, and adopting maximum-difference scaling to further improve annotation quality. They note that word-sense disambiguation and context handling remain necessary before the lexicon reaches full utility.
The main limitations are that annotations reflect prior associations rather than specific contexts and that agreement is lower for emotions such as anticipation. Overall confidence in the core findings is high because of the large sample, multiple validation checks, and agreement with established lexicons.
- Paper: Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks, R. Snow et al. (2008). This foundational paper validates using Amazon Mechanical Turk non-expert crowdsourcing to reliably annotate affect and semantic tasks, establishing the core methodology adopted by the source.
- Paper: Predicting the Semantic Orientation of Adjectives, V. Hatzivassiloglou et al. (1997). This seminal study introduces automated methods to infer semantic orientation and polarity from lexical relations, laying key groundwork for sentiment lexicon construction.
- Paper: Thumbs up? Sentiment Classification using Machine Learning Techniques, Bo Pang et al. (2002). This landmark paper formalizes document-level sentiment classification using machine learning and lexical features, providing the primary application context that motivates large-scale emotion lexicons.
- Paper: Learning Word Vectors for Sentiment Analysis, Andrew L. Maas et al. (2011). It explores learning continuous vector representations that explicitly capture semantic and sentiment polarity information, framing the challenge of representing emotional meaning computationally.
- Paper: Word Association Norms, Mutual Information, and Lexicography, Kenneth Ward Church et al. (1989). This work establishes quantitative measures of word associations and co-occurrence norms, which underpin statistical association studies in lexical semantics.
- Paper: Semantic Similarity in a Taxonomy: An Information-Based Measure and its Application to Problems of Ambiguity in Natural Language, Philip Resnik (1999). It introduces information-theoretic measures of semantic similarity combining taxonomies like WordNet with corpus statistics, directly relevant to the lexical resources referenced by the source.
- Paper: Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification, John Blitzer et al. (2007). This paper highlights the domain dependency and vocabulary shifts inherent in sentiment analysis, motivating broad-coverage lexicons that generalize across contexts.
- Paper: Automated Hate Speech Detection and the Problem of Offensive Language, Thomas Davidson et al. (2017). This study applies crowdsourced annotation and lexical sentiment features to the downstream challenge of distinguishing targeted hate speech from general offensive language.
- Paper: Attention-based LSTM for Aspect-level Sentiment Classification, Yequan Wang et al. (2016). It extends fine-grained sentiment analysis from general word-emotion associations to aspect-level sentiment classification using neural attention mechanisms.
- Paper: Semantics derived automatically from language corpora contain human-like biases, Aylin Caliskan et al. (2016). It demonstrates how human-like affective and semantic associations embedded in language corpora are systematically captured by distributed representations.
- Paper: ConceptNet 5.5: An Open Multilingual Graph of General Knowledge, R. Speer et al. (2016). This paper expands semantic and affective knowledge into a large-scale open multilingual knowledge graph, enriching word representations with relational common-sense data.
- Paper: A large annotated corpus for learning natural language inference, Samuel R. Bowman et al. (2015). It scales up crowdsourcing methodologies to construct the massive Stanford Natural Language Inference corpus for evaluating complex semantic relationships.
