Opinion observer: analyzing and comparing opinions on the Web

Bing LiuMinqing HuJunsheng Cheng

article2005WWW1,892 citations

Presents Opinion Observer, a framework that mines product features from customer reviews using supervised language pattern mining and visually compares consumer sentiments across competing products feature by feature.

Listen

Online customer reviews have become a vital source of market intelligence and consumer guidance, yet analyzing hundreds of unstructured text reviews across competing products is labor-intensive and difficult to scale. Organizations typically rely on costly manual evaluations or broad sentiment scores that obscure why customers favor or dislike specific features.

The article develops and evaluates Opinion Observer, a system designed to extract fine-grained product features from online customer reviews and visually compare consumer sentiment across competing products along specific feature dimensions.

The approach integrates automated web record extraction, natural language processing, and supervised association rule mining to discover language patterns from brief review segments (such as "Pros" and "Cons" sections). The system extracts explicit features, maps values to implicit features, aggregates positive and negative sentiments, and organizes synonyms. Researchers evaluated the framework using customer reviews across 15 electronic products from Epinions.com, training extraction models on 10 products and testing on the remaining five against baseline techniques.

Key findings show that the proposed supervised pattern mining method achieved average recall and precision rates of 90.2% and 88.9% on positive reviews (Pros), and 82.4% and 79.1% on negative reviews (Cons) when combined with a frequent-term refinement strategy. These results markedly outperformed standard baseline methods, including noun-phrase extraction and earlier unsupervised itemset mining, which had Cons precision drop as low as 36.5% and 48.1% respectively. The analysis also revealed that positive feedback generated more distinct linguistic patterns (117 patterns) than negative feedback (22 patterns) due to the greater diversity of language customers use when complaining. In operational testing, the system's semi-automated interface allowed human analysts to correct errors while saving approximately 45% of the time required for purely manual tagging.

These findings indicate that businesses can substantially reduce market research costs, accelerate product benchmarking timelines, and pinpoint engineering or quality vulnerabilities relative to competitors. Rather than relying on aggregate product ratings, decision-makers can view exact consumer sentiment distribution across distinct components such as battery life, software, or build quality.

Organizations should consider deploying automated or semi-automated opinion comparison pipelines to support competitive intelligence, product development, and retail user experiences. For high-stakes decisions requiring near-perfect accuracy, organizations should implement the semi-automated workflow where analysts verify the initial machine-extracted findings.

The study's limitations include a performance drop when analyzing negative feedback due to diverse complaint vocabulary, an inability to resolve context-dependent synonyms (achieving only 52% synonym recall), and an evaluation restricted to consumer electronics. While confidence is high regarding feature extraction from structured pros and cons lists, caution is warranted when applying the system across broader, domain-divergent text sources without supplementary human review.

Cover for Opinion observer: analyzing and comparing opinions on the Web

Abstract

The Web has become an excellent source for gathering consumer opinions. There are now numerous Web sites containing such opinions, e.g., customer reviews of products, forums, discussion groups, and blogs. This paper focuses on online customer reviews of products. It makes two contributions. First, it proposes a novel framework for analyzing and comparing consumer opinions of competing products. A prototype system called Opinion Observer is also implemented. The system is such that with a single glance of its visualization, the user is able to clearly see the strengths and weaknesses of each product in the minds of consumers in terms of various product features. This comparison is useful to both potential customers and product manufacturers. For a potential customer, he/she can see a visual side-by-side and feature-by-feature comparison of consumer opinions on these products, which helps him/her to decide which product to buy. For a product manufacturer, the comparison enables it to easily gather marketing intelligence and product benchmarking information. Second, a new technique based on language pattern mining is proposed to extract product features from Pros and Cons in a particular type of reviews. Such features form the basis for the above comparison. Experimental results show that the technique is highly effective and outperform existing methods significantly.

Table of Contents

  • 1. INTRODUCTION
  • 2. RELATED WORK
  • Terminology finding and entity extraction
  • Sentiment classification
  • 3. OPINION OBSERVER
  • 3.1 Problem Statement
  • 3.2 Visualizing Opinion Comparison
  • 3.3 Automated Opinion Analysis
  • 3.3.1 Extracting Product Features
  • 3.3.2 Grouping Synonyms
  • 3.4 Semi-Automated Tagging of Reviews
  • 3.5 Extracting Reviews from Web Pages
  • 4. SYSTEM ARCHITECTURE
  • 5. EXPERIMENT RESULTS
  • 6. CONCLUSIONS
  • 7. ACKNOWLEDGMENTS
  • 8. REFERENCES

Knowls

  1. Knowl 1 — Supervised Language Pattern Mining for Feature Extraction from Pros and Cons

    model/method

    In product review formats where reviewers write brief Pros and Cons (incomplete sentences or short phrases separated by punctuation such as ,, ., and, and but), product features are extracted using supervised association rule mining:

    1. Pre-processing and Labeling: Review text is tagged with Part-Of-Speech (POS) tags, and specific numeric digits are removed to promote generalization (e.g., converting 16MB to MB). Feature words in the training set are replaced with a generic token [feature]. For implicit features (such as value indicators like MB), the indicator term is replaced with [feature].
    2. nn-Gram Generation: Labeled segments are split into 3-grams consisting of words and their associated POS tags to capture local syntactic structure without generating spurious rules from longer text.
    3. Duplicate POS Indexing and Stemming: Multiple occurrences of the same POS tag in a segment receive sequence indices (e.g., <N1>, <N2>), and words undergo morphological stemming.
    4. Association Rule Mining: Using the CBA rule mining algorithm on the labeled transaction data with a minimum support threshold of 1%1\%, association rules of the form XYX \to Y are generated. Only rules with Y=[feature]Y = [\text{feature}] on the right-hand side are retained.
    5. Pattern Sequencing: The left-hand-side items of each rule are ordered to match the actual word/POS sequence observed in the training text, retaining sequenced rules with a minimum confidence threshold of 50%50\%. These are transformed into extraction patterns (e.g., <N1> [feature] <N2> or easy to <V> [feature]).
    6. Extraction: The resulting patterns are matched against POS-tagged test reviews, allowing word gaps during matching. If no pattern matches a sentence segment, the extractor defaults to extracting nouns or noun phrases identified by the POS tagger.
  2. Knowl 2 — Feature Refinement via Frequent-Term and Frequent-Noun Strategies

    model/method

    To resolve extraction conflicts (multiple candidate features in a single segment) and recover missed genuine features, a frequency-based refinement procedure is applied after initial pattern matching:

    1. All candidate features generated across the entire collection of reviews for a product are compiled into a candidate feature list alongside their occurrence frequencies.
    2. Candidate features that occur more frequently across all reviews of a product are assumed to have a higher likelihood of being genuine product features.
    3. In segments with multiple candidate features or ambiguous extractions, two replacement strategies are used:
      • Frequent-Noun Strategy: If a sentence segment contains two or more nouns, the noun with the highest frequency in the candidate feature list is selected as the segment's feature.
      • Frequent-Term Strategy: For each sentence segment, the term (regardless of its POS tag) with the highest frequency in the candidate feature list is selected as the feature.

    The frequent-term strategy outperforms the frequent-noun strategy because it corrects for POS tagging errors on brief sentence fragments and accommodates non-noun features such as verbs.

  3. Knowl 3 — Visual Bar Height Normalization in Multi-Product Opinion Comparison

    equation

    When visualizing and comparing consumer opinions across a set of selected products SPS \subseteq P along feature dimensions jj, the heights of the positive bar Li,j+L_{i,j}^+ (above the horizontal axis) and the negative bar Li,jL_{i,j}^- (below the horizontal axis) for product iSi \in S on feature jj are normalized by the maximum opinion count across all features and products:

    Li,j+=Ni,j+max(M+,M),Li,j=Ni,jmax(M+,M)L_{i,j}^+ = \frac{N_{i,j}^+}{\max(M^+, M^-)}, \qquad L_{i,j}^- = \frac{N_{i,j}^-}{\max(M^+, M^-)}

    where:

    • Ni,j+N_{i,j}^+ is the count of positive opinion segments (the size of the positive opinion set, PsetPset) for feature jj of product ii.
    • Ni,jN_{i,j}^- is the count of negative opinion segments (the size of the negative opinion set, NsetNset) for feature jj of product ii.
    • Mi+=maxj(Ni,j+)M_i^+ = \max_j (N_{i,j}^+) is the maximum positive opinion count across all features of product ii.
    • Mi=maxj(Ni,j)M_i^- = \max_j (N_{i,j}^-) is the maximum negative opinion count across all features of product ii.
    • M+=maxiS(Mi+)M^+ = \max_{i \in S} (M_i^+) is the maximum positive opinion count across all products in SS and all features.
    • M=maxiS(Mi)M^- = \max_{i \in S} (M_i^-) is the maximum negative opinion count across all products in SS and all features.
  4. Knowl 4 — Evaluation of Pattern-Based Feature Extraction on Pros and Cons

    data/table

    The effectiveness of supervised language pattern mining and post-refinement strategies was evaluated on 5 test electronic products (data1–data5) from Epinions reviews (trained on 10 separate products). Feature extraction was evaluated using per-review recall (rr) and precision (pp):

    r=i=1nECii=1nCi,p=i=1nECii=1nEir = \frac{\sum_{i=1}^n EC_i}{\sum_{i=1}^n C_i}, \qquad p = \frac{\sum_{i=1}^n EC_i}{\sum_{i=1}^n E_i}

    where nn is the total review count, ECiEC_i is the number of correctly extracted features from review ii, CiC_i is the actual feature count in review ii, and EiE_i is the total number of extracted features from review ii.

    Patterns only Frequent-noun Frequent-term
    Pros Recall Prec. Recall Prec. Recall Prec.
    data1 0.878 0.880 0.849 0.861 0.922 0.876
    data2 0.787 0.804 0.798 0.821 0.894 0.902
    data3 0.782 0.806 0.758 0.782 0.825 0.825
    data4 0.943 0.926 0.939 0.926 0.942 0.922
    data5 0.899 0.893 0.878 0.881 0.930 0.923
    Avg. 0.857 0.862 0.844 0.854 0.902 0.889
    Cons Recall Prec. Recall Prec. Recall Prec.
    data1 0.900 0.856 0.867 0.848 0.850 0.798
    data2 0.795 0.794 0.808 0.804 0.860 0.833
    data3 0.677 0.699 0.834 0.801 0.846 0.769
    data4 0.632 0.623 0.654 0.623 0.681 0.657
    data5 0.772 0.772 0.839 0.867 0.881 0.897
    Avg. 0.755 0.748 0.801 0.788 0.824 0.791

    The frequent-term strategy achieves the highest average recall and precision. Performance is higher on Pros than Cons because positive reviews use standardized vocabulary (e.g., 'excellent', 'great', 'good') yielding 117 patterns, whereas negative expressions in Cons are more diverse, yielding only 22 patterns.

  5. Knowl 5 — Baseline Comparison: Language Pattern Mining vs. Noun Phrases and Itemset Mining

    data/table

    The supervised pattern mining approach was compared against two baseline approaches on the 5 electronic product test sets:

    1. Noun/Noun Phrase Baseline: Extracting all nouns and noun phrases identified by POS tagging as candidate features.
    2. FBS Baseline (Feature-Based Summarization): The unsupervised frequent itemset mining method designed for full-sentence reviews.
    Pros Cons
    Noun phrases FBS Noun phrases FBS
    Dataset Recall Prec Recall Prec Recall Prec Recall Prec
    data1 0.543 0.524 0.400 0.476 0.681 0.409 0.419 0.424
    data2 0.747 0.642 0.494 0.567 0.536 0.249 0.485 0.508
    data3 0.551 0.521 0.431 0.508 0.642 0.327 0.486 0.494
    data4 0.728 0.682 0.411 0.441 0.758 0.354 0.496 0.506
    data5 0.664 0.631 0.480 0.560 0.859 0.487 0.469 0.474
    Avg. 0.647 0.600 0.443 0.510 0.700 0.365 0.471 0.481

    Both baseline methods perform poorly on brief Pros/Cons segments. The noun-phrase baseline misses implicit features and non-noun features (verbs, adjectives) and suffers from POS tagger errors on incomplete sentences. FBS fails because unsupervised itemset mining requires full-sentence co-occurrence statistics that do not hold in concise Pros/Cons phrases.

  6. Knowl 6 — Formal Definitions of Product Features, Opinion Segments, and Polarity Sets

    definition

    Let P={P1,P2,,Pn}P = \{P_1, P_2, \dots, P_n\} be a set of products, where each product PiP_i has a set of reviews Ri={r1,r2,,rk}R_i = \{r_1, r_2, \dots, r_k\} and each review rjr_j is a sequence of sentences sj1,sj2,,sjm\langle s_{j1}, s_{j2}, \dots, s_{jm} \rangle.

    • Product Feature (ff): An attribute or component of product PiP_i commented on in review rjr_j. If ff appears literally in rjr_j, it is an explicit feature; if ff is implied without appearing explicitly (e.g., 'too heavy' implies <weight><weight>), it is an implicit feature.
    • Opinion Segment (OsOs): A set of consecutive sentences in review rjr_j expressing a positive or negative opinion about feature ff.
    • Positive Opinion Set (PsetPset): The set of all opinion segments across all reviews RiR_i of product PiP_i that express positive opinions about feature ff.
    • Negative Opinion Set (NsetNset): The set of all opinion segments across all reviews RiR_i of product PiP_i that express negative opinions about feature ff.

    Opinion analysis aggregates PsetPset and NsetNset at two primary levels: Level 1 (overall product-level aggregation across all features) and Level 2 (individual product feature components such as battery, picture, or zoom).

  7. Knowl 7 — Synonym Grouping via Constrained WordNet Synset Filtering

    model/method

    To combine differently worded features referring to the same attribute (e.g., 'photo', 'picture', and 'image'), WordNet synsets are queried with a sense constraint:

    • Unconstrained synonym matching causes domain errors (for instance, treating 'movie' and 'picture' as synonyms in camera reviews conflates video recording capabilities with still photograph quality).
    • Two feature words AA and BB are merged into the same synonym group if and only if there exists a WordNet synset containing both AA and BB that appears within the top two most frequent senses of word AA and the top two most frequent senses of word BB.

    This conservative heuristic achieves 100%100\% precision and 52%52\% recall on product review feature grouping.

  8. Knowl 8 — Semi-Automated Tagging Efficiency Improvement

    empirical result

    In interactive review annotation, human analysts review the automatic extraction output and correct errors (deleting incorrect features, adding missing ones via drop-downs or text inputs, and adjusting sentiment orientation) rather than tagging from scratch.

    In experiments conducted with two human taggers comparing fully manual tagging (reading reviews and manually extracting features and sentiments) with semi-automated tagging (reviewing and correcting automated first-cut outputs using the Opinion Observer interface), the semi-automated procedure reduced total human annotation time by approximately 45%45\% (including time spent reading the reviews).

  9. Knowl 9 — Mapping Value Attributes to Implicit Product Features

    model/method

    Implicit features occur when reviewers describe a property value rather than naming the feature itself (e.g., 'heavy' represents the implicit feature <weight><weight>, 'big' represents <size><size>, and '16MB' represents <memory><memory>).

    Implicit feature resolution is performed during the training and extraction process:

    1. Training data segments are annotated with both the indicator term and its target feature concept (e.g., heavy     weight\implies \langle weight \rangle).
    2. Association rule mining is applied to extract explicit mapping rules connecting value-word descriptors to their respective implicit feature categories.
    3. Extracted indicator words are automatically translated to standard feature categories for aggregated visualization.

Coverage note — The automated Web data record extraction component (MDR-2 / partial tree alignment) was omitted as it is an external prior contribution incorporated as background data ingestion plumbing.

References

  1. 1.Agrawal, R., Rajagopalan, S., Srikant, R., Xu, Y. 2003. Mining newsgroups using networks arising from social behavior. WWW’03.
  2. 2.Agrawal, R. and Srikant, R. 1994. Fast algorithm for mining association rules. VLDB’94.
  3. 3.Aoki, P. and Woodruff, A. 2004. “User Interfaces” and the Social Negotiation of Availability. ACM SIGCHI Workshop on Forecasting Presence and Availability.
  4. 4.Bourigault, D. 1995. Lexter: A terminology extraction software for knowledge acquisition from texts. KAW’95.
  5. 5.Bunescu, R., Mooney, R. 2004. Collective Information Extraction with Relational Markov Networks. ACL’2004.
  6. 6.Chi, E., Hong, L., Heiser, J., Card, S. 2004. eBooks with Indexes that Reorganize Conceptually. Proc. of CHI2004 Conference Companion.
  7. 7.Daille, B. 1996. Study and Implementation of Combined Techniques for Automatic Extraction of Terminology. The Balancing Act: Combining Symbolic and Statistical Approaches to Language. MIT Press, Cambridge
  8. 8.Das, S. and Chen, M., 2001. Yahoo! for Amazon: Extracting market sentiment from stock message boards. APFA’01.
  9. 9.Dave, K., Lawrence, S., and Pennock, D. 2003. Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of Product Reviews. WWW’03.
  10. 10.Etzioni, O., Cafarella, M., Downey, D., Kok, S. Popescu, A., Shaked, T., Soderland, S., Weld, S. Web-Scale Information Extraction in KnowItAll (Preliminary Results). WWW’2004.
  11. 11.FASTR. http://www.limsi.fr/Individu/jacquemi/FASTR/
  12. 12.Fellbaum, C. 1998. WordNet: an Electronic Lexical Database, MIT Press.
  13. 13.Freire, J., Kumar, B. and Lieuwen, D. 2001. WebViews: Accessing personalized Web content and services. WWW’10.
  14. 14.Freitag, D., McCallum, A. 2000. Information extraction with HMM structures learned by stochastic optimization. AAAI’00.
  15. 15.Hatzivassiloglou, V., and Wiebe, J. 2000. Effects of adjective orientation and gradability on sentence subjectivity. COLING’00.
  16. 16.Hearst, M. 1992. Direction-based Text Interpretation as an Information Access Refinement. In P. Jacobs, editor, Text-Based Intelligent Systems. Lawrence Erlbaum Associates.
  17. 17.Hu, M., and Liu, B. 2004. Mining and summarizing customer reviews. KDD’04, 2004.
  18. 18.Jacquemin, C., Bourigault, D. 2001. Term extraction and automatic indexing. In R. Mitkov, editor, Handbook of Computational Linguistics. Oxford University Press.
  19. 19.Justeson, J., Katz, S. 1995. Technical Terminology: some linguistic properties and an algorithm for identification in text. Natural Language Engineering 1(1):9-27.
  20. 20.Lafferty, J., McCallum, A., Pereira, F. 2001. Conditional random fields: probabilistic models for segmenting and labeling or sequence data. ICML’01.
  21. 21.Liu, B., Hsu, W., Ma, Y. 1998. Integrating Classification and Association Rule Mining. KDD’98, 1998.
  22. 22.Liu, B., Grossman, R., and Zhai, Y. “Mining data records from Web pages.” KDD’03, 2003.
  23. 23.Morinaga, S., Yamanishi, K., Tateishi, K., and Fukushima, T. 2002. Mining Product Reputations on the Web. KDD’02.
  24. 24.Muslea, I., Minton, S. and Knoblock, C. “A hierarchical approach to wrapper induction.” Agents’99, 1999.
  25. 25.MySql, http://www.mysql.com/
  26. 26.Nasukawa, T. and Yi, J. 2003. Sentiment analysis: Capturing favorability using natural language processing. Proceedings of the 2nd Intl. Conf. on Knowledge Capture (K-CA’2003).
  27. 27.Nigam, K., and Hurst, M. Towards a Robust Metric of Opinion. AAAI Spring Symposium on Exploring Attitude and Affect in Text. 2004.
  28. 28.NLProcessor – Text Analysis Toolkit. 2000. http://www.infogistics.com/textanalysis.html
  29. 29.Pang, B., Lee, L., and Vaithyanathan, S., 2002. Thumbs up? Sentiment Classification Using Machine Learning Techniques. EMNLP’2002.
  30. 30.Riloff, E. and Wiebe, J. 2003. Learning extraction patterns for subjective expressions. EMNLP’2003.
  31. 31.Rosario, B., and Hearst, M., Classifying Semantic Relations in Bioscience Text. ACL’2004.
  32. 32.Tong, R. 2001. An Operational System for Detecting and Tracking Opinions in on-line discussion. SIGIR 2001 Workshop on Operational Text Classification.
  33. 33.Turney, P. 2002. Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews. ACL’02.
  34. 34.Wiebe, J., Bruce, R., and O’Hara, T. 1999. Development and Use of a Gold Standard Data Set for Subjectivity Classifications. ACL’99.
  35. 35.Wilson, T., Wiebe, J., and Hwa, R. 2004. Just how mad are you? Finding strong and weak opinion clauses. AAAI’04.
  36. 36.Yu, H., and Hatzivassiloglou, V. 2003. Towards answering opinion questions: Separating facts from opinions and identifying the polarity of opinion sentences. EMNLP’03.
  37. 37.Zhai, Y., and Liu, B. Web data extraction based on partial tree alignment. WWW’05, 2005.

Citation

MLA
Liu, B., et al. “Opinion Observer”. Proceedings of the 14th International Conference on World Wide Web - WWW '05, 2005, p. 342, https://doi.org/10.1145/1060745.1060797.
APA
Liu, B., Hu, M., & Cheng, J. (2005). Opinion observer. Proceedings of the 14th International Conference on World Wide Web - WWW '05, 342. https://doi.org/10.1145/1060745.1060797
Chicago
Liu, B., M. Hu, and J. Cheng. 2005. “Opinion Observer”. Proceedings of the 14th International Conference on World Wide Web - WWW '05, 342. https://doi.org/10.1145/1060745.1060797.
Harvard
Liu, B., Hu, M. and Cheng, J. (2005) “Opinion observer”, Proceedings of the 14th international conference on World Wide Web - WWW '05. ACM Press, p. 342. Available at: https://doi.org/10.1145/1060745.1060797.
Vancouver
1. Liu B, Hu M, Cheng J (2005) Opinion observer. In: Proceedings of the 14th international conference on World Wide Web - WWW '05. ACM Press, p 342

BibTeX

@inproceedings{Liu_2005, series={WWW ’05}, title={Opinion observer: analyzing and comparing opinions on the Web}, url={http://dx.doi.org/10.1145/1060745.1060797}, DOI={10.1145/1060745.1060797}, booktitle={Proceedings of the 14th international conference on World Wide Web  - WWW ’05}, publisher={ACM Press}, author={Liu, Bing and Hu, Minqing and Cheng, Junsheng}, year={2005}, pages={342}, collection={WWW ’05} }
Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF