Determining the Sentiment of Opinions

article2004COLING1,757 citations

Develops a topic-oriented sentiment analysis framework that identifies opinion holders and computes sentence-level polarity by combining WordNet-derived word sentiment strengths within targeted syntactic regions.

Listen

Understanding public sentiment across large volumes of text is essential for tracking organizational reputation, policy debates, and public discourse. However, identifying who holds an opinion and determining whether that opinion is positive or negative remains a difficult automated task due to the complex ways people express emotions and claims in natural language.

The main objective of the article is to demonstrate an automated system that identifies the specific holders of opinions on a given topic and accurately classifies the polarity of their sentiment as positive, negative, or neutral.

The approach operates in stages, combining word-level sentiment evaluation with sentence-level analysis. The system builds lexical resources by expanding a small hand-curated seed list of positive and negative verbs and adjectives using synonyms and antonyms from an established linguistic database. It then detects opinion holders using an automated entity recognizer, defines specific text regions between the holder and the topic, and combines individual word polarities using statistical and algebraic models. The system was evaluated using test sets of 462 adjectives, 502 verbs, and 100 topic-focused news sentences.

The findings show that expanding the initial seed words achieved strong coverage, enabling the classification of over 93% of adjectives and 83% of verbs, which improved further when additional annotated words were incorporated. At the sentence level, the system achieved 81% accuracy when the opinion holder was identified manually, but dropped to 67% accuracy when using automated holder detection—a decline of about 14 percentage points, misclassifying roughly 11% more sentences. Simpler combination models that tracked the presence and reversal of negative words generally matched or outperformed more complex probabilistic averaging models, and focusing on a text window extending from the holder to the end of the sentence yielded the best performance.

These results indicate that automated opinion mining can achieve viable baseline performance without requiring massive, costly human-annotated training corpora. However, the drop in performance between manual and automated holder detection underscores that correctly attributing an opinion to the right entity is a major performance bottleneck, creating risks of misattributing stances in multi-party or syntactically complex sentences.

To improve performance before deploying this technology in high-stakes operational environments, future efforts should integrate syntactic parsers to map grammatical relationships between holders and sentiment phrases rather than relying solely on word proximity. Incorporating multi-word phrases and advanced machine learning algorithms will also be necessary to capture nuanced context.

Confidence in the system's foundational methodology is moderate to high, but decision-makers should exercise caution regarding its current limitations. The evaluation relied on a relatively small sentence sample, and the current models struggle to interpret implicit sentiment expressed through factual statements or to differentiate truly neutral opinions from purely non-opinionated statements.

Information Sciences Institute University of Southern California part of speech affective state semantic orientation sentence sentiment classifier sentiment-bearing words sentiment polarity

Kim et al (2004).pdf

Paper: Thumbs up? Sentiment Classification using Machine Learning Techniques, Bo Pang et al. (2002). This foundational paper establishes the standard machine learning formulations and baselines for document-level sentiment classification that the source adapts and compares against.
Paper: Measuring praise and criticism: Inference of semantic orientation from association, Peter D. Turney et al. (2003). It introduces unsupervised statistical association techniques for inferring word-level semantic orientation, which directly underpins the source's word sentiment determination module.
Paper: Predicting the Semantic Orientation of Adjectives, Vasileios Hatzivassiloglou et al. (1997). It provides the seminal linguistic methodology for extracting the semantic orientation of adjectives from syntactic conjunctions, serving as a key precursor to word-level sentiment modeling.
Paper: Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews, Peter D. Turney (2002). It demonstrates how to aggregate phrase-level semantic orientations into review classifications, providing an essential template for sentence- and document-level sentiment combination.
Paper: Mining the peanut gallery: opinion extraction and semantic classification of product reviews, Kushal Dave et al. (2003). It introduces techniques for opinion extraction and sentiment classification at the sentence level across product reviews, directly anticipating the source's fine-grained opinion analysis.

Paper: A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts, Bo Pang et al. (2004). This work extends sentence-level sentiment and subjectivity processing by formulating context-aware sentence extraction through graph min-cuts.
Paper: Mining and summarizing customer reviews, Minqing Hu et al. (2004). It expands upon sentence sentiment combination by linking opinion orientations directly to specific product features mined from unstructured customer reviews.
Paper: Opinion observer: analyzing and comparing opinions on the Web, Bing Liu et al. (2005). It develops a complete feature-based opinion mining and comparison system, operationalizing sentence-level sentiment extraction for comparative analysis on the web.
Paper: Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales, Bo Pang et al. (2005). It extends discrete sentiment classification to multi-point rating scales using metric labeling and sentence-level polarity distributions.
Paper: Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification, John Blitzer et al. (2007). It investigates cross-domain adaptation for sentiment classifiers, addressing the domain-specificity of word- and sentence-level sentiment indicators.
Paper: Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank, Richard Socher et al. (2013). It generalizes syntactic and semantic compositionality for sentiment by modeling phrase-to-sentence sentiment aggregation with recursive neural tensor networks.
Paper: Attention-based LSTM for Aspect-level Sentiment Classification, Yequan Wang et al. (2016). It advances target-specific sentiment identification by employing neural attention mechanisms over sentence context to capture aspect-level polarities.
Paper: Deep learning for sentiment analysis: A survey, Lei Zhang et al. (2018). It provides a comprehensive survey tracing the evolution from early lexicon and sentence-composition sentiment models to modern deep learning architectures.

Cover for Determining the Sentiment of Opinions

Abstract

Identifying sentiments (the affective parts of opinions) is a challenging problem. We present a system that, given a topic, automatically finds the people who hold opinions positive about that topic and the sentiment of each opinion. The system contains a module for determining word sentiment and another for combining sentiments within a sentence. We experiment with various models of classifying and combining sentiment at word and sentence levels, with promising results.

Topics

part of speech affective state semantic orientation

Keywords

sentence sentiment classifier sentiment-bearing words sentiment polarity semantic orientation opinion holder automatic holder detection

1 Introduction
2 Algorithm
2.1 Word Sentiment Classifier
2.1.1 Word Classification Models
2.2 Sentence Sentiment Classifier
2.2.1 Holder Identification
2.2.2 Sentiment Region
2.2.3 Classification Models
2.2.4 Examples
3 Experiments
3.1 Word Sentiment Classifier
3.1.2 Human-Machine Agreement
3.2 Sentence Sentiment Classifier
3.2.1 Data
3.2.2 Test on Human Annotated Data
3.3 Problems
3.3.1 Word Sentiment Classification
3.3.2 Sentence Sentiment Classification
3.4 Discussion
4 Conclusion
References

Knowls

Knowl 1 — Topic-Oriented Opinion Sentiment and Holder Extraction Pipeline
algorithm
The opinion sentiment extraction pipeline identifies the sentiment polarity (positive, negative, or neutral) held by specific entities toward a predefined topic within unstructured text across four stages:
Input: Target topic phrase $T$ , input document collection $D$
Output: Tuples of (opinion region, holder $H$ , sentiment polarity $C$ )
for each sentence $s \in D$ do
    Identify named entity candidates (PERSON and ORGANIZATION) in $s$
    if $s$ contains $T$ and at least one entity candidate then
        Select holder $H$ as the candidate entity nearest in word distance to $T$
        Delimit a sentiment region window $R \subseteq s$ relative to $H$ and $T$
        for each sentiment-bearing word $w \in R$ do
            Determine word polarity probabilities $P(c|w)$ for $c \in \{\text{positive}, \text{negative}\}$ using WordNet synonym expansions
        end for
        Aggregate individual word sentiments in $R$ into sentence polarity $C \in \{\text{positive}, \text{negative}\}$
        return $(R, H, C)$
    end if
end for
Knowl 2 — Word Sentiment Classification via WordNet Seed Expansion and Synonym Probability Models
equation
Given a seed lexicon initialized with known positive and negative adjectives and verbs, the lexicon is expanded across multiple cycles via WordNet synonyms (and antonyms for adjectives, inverted into opposite categories). For an unseen word $w$ , sentiment polarity $c \in \{\text{positive}, \text{negative}\}$ is classified using one of two probability formulations over its WordNet synonym set $\text{synset}(w) = \{\text{syn}_1, \text{syn}_2, \dots, \text{syn}_n\}$ :
1. Document classification-derived model (using feature words $f_k$ in sentiment class $c$ ): $P(c|w) \propto P(c) \prod_{k=1}^m P(f_k|c)^{\text{count}(f_k, \text{synset}(w))}$ where $P(c)$ is the proportion of total seed words belonging to class $c$ , $f_k$ is the $k$ -th feature word in sentiment class list $c$ that is also in $\text{synset}(w)$ , and $\text{count}(f_k, \text{synset}(w))$ is the count of occurrences of $f_k$ in the synonym set of $w$ .
2. Direct synonym overlap model: $P(c|w) \propto P(c) \frac{\sum_{i=1}^n \text{count}(\text{syn}_i, c)}{\text{count}(c)}$ where $\text{count}(\text{syn}_i, c)$ is $1$ if $\text{syn}_i$ occurs in class list $c$ and $0$ otherwise, and $\text{count}(c)$ is the total word count in class list $c$ .
Knowl 3 — Sentence Sentiment Aggregation Models
model/method
To combine the polarities and strengths of individual sentiment-bearing words within a delimited sentence region $R$ into an overall sentiment category for an opinion holder, three aggregation models are defined:
- Model 0 (Sign Multiplication with Negation): $\text{Polarity}(s) = \prod_{w \in R} \text{sign}(w)$ where $\text{sign}(w) \in \{+1, -1\}$ represents the polarity class of word $w$ . Negation words (e.g., not, never) within the region reverse the polarity sign.
- Model 1 (Averaged Sentiment Strength): $P(c|s) = \frac{1}{n(c)} \sum_{i=1}^{n(c)} P(c|w_i)$ where $w_i \in R$ are words for which class $c = \arg\max_j P(c_j|w_i)$ , and $n(c)$ is the total number of words in region $R$ assigned to class $c$ .
- Model 2 (Geometric Sentiment Strength): $P(c|s) = 10^{n(c) - 1} \prod_{i=1}^{n(c)} P(c|w_i)$ where $w_i \in R$ are words for which class $c = \arg\max_j P(c_j|w_i)$ , and $n(c)$ is the count of words in region $R$ assigned to class $c$ .
Knowl 4 — Sentiment Region Window Definitions
model/method
Because sentiment expressions are most reliably associated with opinion holders and topics when positioned near them, sentiment-bearing words are aggregated only within a constrained token window $R$ in the sentence:
- Window 1 (Full sentence): The entire sentence containing the holder and topic.
- Window 2 (Between Holder and Topic): Only the tokens occurring strictly between the identified Holder entity and the Topic phrase.
- Window 3 (Window 2 with padding): The token span between Holder and Topic expanded by $\pm 2$ tokens at each boundary.
- Window 4 (Holder to sentence end): The token span starting from the Holder entity and continuing to the end of the sentence.
Knowl 5 — Opinion Quadruple Representation
definition
An opinion is defined as a 4-tuple: $[\text{Topic}, \text{Holder}, \text{Claim}, \text{Sentiment}]$ where:
- $\text{Topic}$ is the primary subject or question under consideration.
- $\text{Holder}$ is the person or organization who maintains the belief.
- $\text{Claim}$ is the proposition or assertion believed by the Holder about the Topic.
- $\text{Sentiment}$ is the explicit or implicit expression of the Holder's positive, negative, or neutral affective regard or desire toward the Claim about the Topic.
Factual claims or beliefs lacking affective evaluation (such as "The Gap is likely to go bankrupt") possess a Topic, Holder, and Claim, but no Sentiment.

Knowl 6 — Word Sentiment Classification Evaluation on Adjectives and Verbs

data/table

The word sentiment classifier was evaluated on common English words (intersected from a TOEFL word list with adjective and verb vocabularies) against human annotation. Lenient agreement evaluates binary negative versus non-negative (positive and neutral merged) accuracy:

Configuration	Adjectives			Verbs
	H1:Machine	H2:Machine	Recall	H1:Machine	H3:Machine	Recall
Random baseline (10 iter avg)	59.35%	57.81%	100%	59.02%	56.59%	100%
Basic seed method	68.37%	68.60%	93.07%	75.84%	72.72%	83.27%
Expanded training data	75.66%	77.88%	97.84%	81.20%	79.06%	93.23%

The basic seed method used 34 seed adjectives (15 positive, 19 negative) and 44 seed verbs (23 positive, 21 negative). The expanded configuration added 231 manually labeled adjectives and 251 verbs to the training set, which increased both agreement with human judges and vocabulary coverage (recall).

Knowl 7 — Sentence-Level Sentiment Accuracy and Impact of Automatic Holder Detection
empirical result
Evaluated on 100 sentences from the DUC 2001 corpus covering four political topics (illegal alien, term limits, gun control, and NAFTA), human inter-annotator agreement on three-way sentiment tagging achieved Cohen's $\kappa = 0.91$ .

When evaluated against human annotations:
- With manually tagged (gold) opinion holders, the highest-performing sentence sentiment combination model achieved 81% accuracy (compared to a majority/random baseline of ~51.5% to ~54.0%).
- With automatic holder detection using named entity tagging and closest-entity heuristics, peak accuracy dropped to 67%.
- Across 40 model/window configurations, automatic holder detection introduced an average misclassification penalty of approximately 7 additional sentences (an absolute error increase of roughly 11 percentage points: 7.060 sentences for Annotator 1 and 6.698 sentences for Annotator 2).
Knowl 8 — Comparative Effectiveness of Sentiment Windows and Combination Rules
empirical result
Systematic comparison across 4 sentence combination models, 4 region window definitions, and 4 word-level probability variants revealed that:
1. Combination model: Model 0 (sign multiplication with negation flipping) achieved the highest individual sentence accuracy, showing that detecting the presence and parity of negative words is more critical than accumulating continuous probability strengths.
2. Region size: When the holder and topic are identified, Window 4 (spanning from the Holder to the end of the sentence) consistently outperformed the full sentence (Window 1) and bounded spans (Windows 2 and 3).
Knowl 9 — Limitations of Lexicon-Based Sentiment and Heuristic Holder Extraction
limitation
The approach exhibits four key failure modes:
1. Implicit / Factual Sentiment: The system cannot infer sentiment conveyed via purely factual assertions lacking affective words (e.g., "She thinks term limits will give women more opportunities in politics").
2. Context-Dependent Word Sentiment: Polysemous words or multi-word idioms (e.g., "hit at democracy") convey sentiment that unigram WordNet synonym lookup fails to assign reliably without wider context or n-gram handling.
3. Multiple Opinions and Holders: In sentences expressing multiple conflicting viewpoints or containing multiple named entities, the proximity heuristic (selecting the holder nearest the topic) frequently links sentiment to the wrong agent.
4. Neutral vs. Non-Opinion Discrimination: Distinguishing neutral opinions from objective non-opinion statements remains unreliable without full syntactic parsing and discourse-level subjectivity modeling.

Coverage note — None was omitted; all contributed models (word and sentence level), region definitions, mathematical equations, empirical evaluations, and stated limitations are fully covered.

References

1.Aristotle. The Rhetorics and Poetics (trans. W. Rhys Roberts), Modern Library, 1954.
2.Fellbaum, C., D. Gross, and K. Miller. 1993. Adjectives in WordNet. http://www.cosgi.princeton.edu/~wn.
3.Hatzivassiloglou, V. and K. McKeown 1997. Predicting the Semantic Orientation of Adjectives. Proceedings of the 35th ACL conference, 174–181.
4.Miller, G.A., R. Beckwith, C. Fellbaum, D. Gross, and K. Miller. 1993. Introduction to WordNet: An On-Line Lexical Database. http://www.cosgi.princeton.edu/~wn.
5.Pang, B. L. Lee, and S. Vaithyanathan, 2002. Thumbs up? Sentiment classification using Machine Learning Techniques. Proceedings of the EMNLP conference.
6.Perelman, C. 1970. The New Rhetoric: A Theory of Practical Reasoning. In The Great Ideas Today. Chicago: Encyclopedia Britannica.
7.Riloff, E., J. Wiebe, and T. Wilson 2003. Learning Subjective Nouns Using Extraction Pattern Bootstrapping. Proceedings of the CoNLL-03 conference.
8.Siegel, S. and N.J. Castellan Jr. 1988. Nonparametric Statistics for the Behavioral Sciences. McGraw-Hill.
9.Toulmin, S.E., R. Rieke, and A. Janik. 1979. An Introduction to Reasoning. Macmillan, New York.
10.Toulmin, S.E. 2003. The Uses of Argument. Cambridge University Press.
11.Turney, P. 2002. Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews. Proceedings of the 40th Annual Meeting of the ACL, Philadelphia, 417–424.
12.Wallace, K. 1975. Topoi and the Problem of Invention. In W. Ross Winterowd (ed), Contemporary Rhetoric. Harcourt Brace Jovanovich.
13.Wiebe, J. et al. 2002. NRRC summer study Jan Wiebe and group (University of Pittsburgh) on ‘subjective’ statements.
14.Yu, H. and V. Hatzivassiloglou. 2003. Towards Answering Opinion Questions: Separating Facts from Opinions and Identifying the Polarity of Opinion Sentences. Proceedings of the EMNLP conference.

Citation

MLA

Kim, S.-M., and E. Hovy. “Determining the Sentiment of Opinions”. COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, 2004, pp. 1367–73, https://aclanthology.org/C04-1200/.

APA

Kim, S.-M., & Hovy, E. (2004). Determining the Sentiment of Opinions. COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, 1367–1373. https://aclanthology.org/C04-1200/

Chicago

Kim, S.-M., and E. Hovy. 2004. “Determining the Sentiment of Opinions”. COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, 1367–73. https://aclanthology.org/C04-1200/.

Harvard

Kim, S.-M. and Hovy, E. (2004) “Determining the Sentiment of Opinions”, COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics. Association for Computational Linguistics, pp. 1367–1373. Available at: https://aclanthology.org/C04-1200/.

Vancouver

1. Kim S-M, Hovy E (2004) Determining the Sentiment of Opinions. In: COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics. Association for Computational Linguistics, pp 1367–1373

BibTeX

@inproceedings{kim-hovy-2004-determining,
    title = "Determining the Sentiment of Opinions",
    author = "Kim, Soo-Min  and
      Hovy, Eduard",
    booktitle = "{COLING} 2004: Proceedings of the 20th International Conference on Computational Linguistics",
    month = "aug 23–aug 27",
    year = "2004",
    address = "Geneva, Switzerland",
    publisher = "COLING",
    url = "https://aclanthology.org/C04-1200/",
    pages = "1367--1373"
}

Metadata:ACL Anthology

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Determining the Sentiment of Opinions

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Topic-Oriented Opinion Sentiment and Holder Extraction Pipeline

Knowl 2 — Word Sentiment Classification via WordNet Seed Expansion and Synonym Probability Models

Knowl 3 — Sentence Sentiment Aggregation Models

Knowl 4 — Sentiment Region Window Definitions

Knowl 5 — Opinion Quadruple Representation

Knowl 6 — Word Sentiment Classification Evaluation on Adjectives and Verbs

Knowl 7 — Sentence-Level Sentiment Accuracy and Impact of Automatic Holder Detection

Knowl 8 — Comparative Effectiveness of Sentiment Windows and Combination Rules

Knowl 9 — Limitations of Lexicon-Based Sentiment and Heuristic Holder Extraction

References

Citation

BibTeX

Access the Paper